[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 538695012 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002361] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009017] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010017] pid_max: default: 32768 minimum: 301 [ 0.011154] LSM: Security Framework initializing [ 0.012047] Yama: becoming mindful. [ 0.013030] SELinux: Initializing. [ 0.014052] *** VALIDATE selinux *** [ 0.023751] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028483] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029222] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031105] *** VALIDATE tmpfs *** [ 0.033058] *** VALIDATE proc *** [ 0.034000] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.036206] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037213] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039026] Spectre V2 : User space: Vulnerable [ 0.040007] Speculative Store Bypass: Vulnerable [ 0.043327] debug: unmapping init [mem 0xffffffffbd059000-0xffffffffbd060fff] [ 0.045920] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046613] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047021] ... version: 2 [ 0.048011] ... bit width: 48 [ 0.049009] ... generic registers: 4 [ 0.050009] ... value mask: 0000ffffffffffff [ 0.051011] ... max period: 00007fffffffffff [ 0.052009] ... fixed-purpose events: 3 [ 0.053008] ... event mask: 000000070000000f [ 0.054272] rcu: Hierarchical SRCU implementation. [ 0.056413] smp: Bringing up secondary CPUs ... [ 0.057522] x86: Booting SMP configuration: [ 0.058020] .... node #0, CPUs: #1 #2 #3 [ 0.066467] smp: Brought up 1 node, 4 CPUs [ 0.068008] smpboot: Max logical packages: 1 [ 0.069010] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.111000] node 0 deferred pages initialised in 40ms [ 0.115006] devtmpfs: initialized [ 0.116189] x86/mm: Memory block size: 128MB [ 0.119034] gcov: version magic: 0x41383552 [ 0.121183] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.122086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.123229] pinctrl core: initialized pinctrl subsystem [ 0.124212] [ 0.124971] ************************************************************* [ 0.125020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.126012] ** ** [ 0.127014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.128010] ** ** [ 0.129012] ** This means that this kernel is built to expose internal ** [ 0.130010] ** IOMMU data structures, which may compromise security on ** [ 0.131011] ** your system. ** [ 0.132030] ** ** [ 0.133010] ** If you see this message and you are not debugging the ** [ 0.134018] ** kernel, report this immediately to your vendor! ** [ 0.135011] ** ** [ 0.136010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.137009] ************************************************************* [ 0.138639] NET: Registered protocol family 16 [ 0.139400] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.140050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.141131] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.142818] cpuidle: using governor menu [ 0.143742] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.144405] PCI: Using configuration type 1 for base access [ 0.145118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.152107] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.153032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.154527] cryptd: max_cpu_qlen set to 1000 [ 0.155224] ACPI: Added _OSI(Module Device) [ 0.157065] ACPI: Added _OSI(Processor Device) [ 0.159011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.160010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.165772] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.173726] ACPI: Interpreter enabled [ 0.174193] ACPI: PM: (supports S0 S3 S4 S5) [ 0.175009] ACPI: Using IOAPIC for interrupt routing [ 0.177126] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.181423] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.193826] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.196036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.199015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.203142] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.209545] acpiphp: Slot [2] registered [ 0.211103] acpiphp: Slot [5] registered [ 0.212259] acpiphp: Slot [6] registered [ 0.214208] acpiphp: Slot [7] registered [ 0.216102] acpiphp: Slot [8] registered [ 0.217088] acpiphp: Slot [9] registered [ 0.219142] acpiphp: Slot [10] registered [ 0.221106] acpiphp: Slot [3] registered [ 0.222156] acpiphp: Slot [4] registered [ 0.224106] acpiphp: Slot [11] registered [ 0.225077] acpiphp: Slot [12] registered [ 0.227084] acpiphp: Slot [13] registered [ 0.228128] acpiphp: Slot [14] registered [ 0.230111] acpiphp: Slot [15] registered [ 0.232132] acpiphp: Slot [16] registered [ 0.234132] acpiphp: Slot [17] registered [ 0.236114] acpiphp: Slot [18] registered [ 0.237126] acpiphp: Slot [19] registered [ 0.239108] acpiphp: Slot [20] registered [ 0.241101] acpiphp: Slot [21] registered [ 0.242099] acpiphp: Slot [22] registered [ 0.244099] acpiphp: Slot [23] registered [ 0.246097] acpiphp: Slot [24] registered [ 0.247093] acpiphp: Slot [25] registered [ 0.249074] acpiphp: Slot [26] registered [ 0.250186] acpiphp: Slot [27] registered [ 0.252106] acpiphp: Slot [28] registered [ 0.253074] acpiphp: Slot [29] registered [ 0.255083] acpiphp: Slot [30] registered [ 0.256065] acpiphp: Slot [31] registered [ 0.258049] PCI host bridge to bus 0000:00 [ 0.260014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.262018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.265020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.267026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.270015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.273018] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.275188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.280358] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.284424] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.295880] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.300051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.303016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.306016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.309016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.313189] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.315845] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.318042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.322009] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.329013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.344013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.350012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.357938] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.367012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.374000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.407017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.418102] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.425013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.431012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.474015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.490698] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.510030] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.521018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.547066] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.561253] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.573014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.582014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.603014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.612221] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.618013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.624011] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.647014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.658758] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.665025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.672012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.690014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.705583] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.711419] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.714640] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.719574] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.722278] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.729061] iommu: Default domain type: Passthrough [ 0.732210] SCSI subsystem initialized [ 0.735224] ACPI: bus type USB registered [ 0.737641] usbcore: registered new interface driver usbfs [ 0.742105] usbcore: registered new interface driver hub [ 0.745086] usbcore: registered new device driver usb [ 0.748279] pps_core: LinuxPPS API ver. 1 registered [ 0.750009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.754167] PTP clock support registered [ 0.757072] EDAC MC: Ver: 3.0.0 [ 0.759190] PCI: Using ACPI for IRQ routing [ 0.763745] NetLabel: Initializing [ 0.765007] NetLabel: domain hash size = 128 [ 0.767009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.770125] NetLabel: unlabeled traffic allowed by default [ 0.773559] vgaarb: loaded [ 0.775457] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.778068] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.788433] clocksource: Switched to clocksource kvm-clock [ 0.981140] VFS: Disk quotas dquot_6.6.0 [ 0.983109] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.985916] *** VALIDATE ramfs *** [ 0.987232] *** VALIDATE hugetlbfs *** [ 0.988953] pnp: PnP ACPI init [ 0.991532] pnp: PnP ACPI: found 6 devices [ 1.008808] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.012461] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.014705] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.017317] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.020163] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.022825] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.026355] NET: Registered protocol family 2 [ 1.029079] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.034396] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.039592] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.045813] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.049580] TCP: Hash tables configured (established 65536 bind 65536) [ 1.053300] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.057633] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.147678] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.154077] NET: Registered protocol family 1 [ 1.159311] RPC: Registered named UNIX socket transport module. [ 1.162141] RPC: Registered udp transport module. [ 1.163964] RPC: Registered tcp transport module. [ 1.166173] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.169035] NET: Registered protocol family 44 [ 1.170899] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.173450] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.175934] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.178552] PCI: CLS 0 bytes, default 64 [ 1.180427] Unpacking initramfs... [ 3.090942] debug: unmapping init [mem 0xffff9ed27cc54000-0xffff9ed27ffbffff] [ 3.094736] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.142381] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.145978] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.811132] Initialise system trusted keyrings [ 3.813258] Key type blacklist registered [ 3.870597] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.881958] zbud: loaded [ 3.885272] *** VALIDATE nfs *** [ 3.886574] *** VALIDATE nfs4 *** [ 3.888267] pstore: using deflate compression [ 3.893193] Platform Keyring initialized [ 4.036735] NET: Registered protocol family 38 [ 4.039488] Key type asymmetric registered [ 4.041554] Asymmetric key parser 'x509' registered [ 4.044676] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.048936] io scheduler mq-deadline registered [ 4.051981] io scheduler kyber registered [ 4.054765] io scheduler bfq registered [ 4.056957] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.060979] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.065712] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.073694] ACPI: Power Button [PWRF] [ 4.093196] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.103482] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.130506] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.141480] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.167268] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.199662] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.230848] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.236469] Non-volatile memory driver v1.3 [ 4.238635] Linux agpgart interface v0.103 [ 4.273447] virtio_blk virtio1: [vda] 134008 512-byte logical blocks (68.6 MB/65.4 MiB) [ 4.277905] vda: detected capacity change from 0 to 68612096 [ 4.292394] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.295871] vdb: detected capacity change from 0 to 1073741824 [ 4.313257] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.317246] vdc: detected capacity change from 0 to 2621440000 [ 4.333304] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.341122] vdd: detected capacity change from 0 to 2621440000 [ 4.373623] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.377329] vde: detected capacity change from 0 to 4294967296 [ 4.406272] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.410233] vdf: detected capacity change from 0 to 4294967296 [ 4.419079] libphy: Fixed MDIO Bus: probed [ 4.453722] usbcore: registered new interface driver usbserial_generic [ 4.463729] usbserial: USB Serial support registered for generic [ 4.468318] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.472889] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.475134] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.478275] mousedev: PS/2 mouse device common for all mice [ 4.483379] rtc_cmos 00:05: RTC can wake from S4 [ 4.483453] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.488854] rtc_cmos 00:05: registered as rtc0 [ 4.503705] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.508405] intel_pstate: CPU model not supported [ 4.514811] hid: raw HID events driver (C) Jiri Kosina [ 4.517995] usbcore: registered new interface driver usbhid [ 4.519411] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.524240] usbhid: USB HID core driver [ 4.549245] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.551844] drop_monitor: Initializing network drop monitor service [ 4.558340] Initializing XFRM netlink socket [ 4.562034] NET: Registered protocol family 10 [ 4.565472] Segment Routing with IPv6 [ 4.567158] NET: Registered protocol family 17 [ 4.569271] mpls_gso: MPLS GSO support [ 4.575665] RAS: Correctable Errors collector initialized. [ 4.578124] AVX version of gcm_enc/dec engaged. [ 4.579983] AES CTR mode by8 optimization enabled [ 4.761740] sched_clock: Marking stable (4761519368, 0)->(5781087356, -1019567988) [ 4.767425] registered taskstats version 1 [ 4.771361] Loading compiled-in X.509 certificates [ 4.778909] zswap: loaded using pool lzo/zbud [ 4.824243] Key type big_key registered [ 4.841168] Key type encrypted registered [ 4.843462] ima: No TPM chip found, activating TPM-bypass! [ 4.845957] ima: Allocated hash algorithm: sha1 [ 4.847901] ima: No architecture policies found [ 4.849899] evm: Initialising EVM extended attributes: [ 4.852445] evm: security.selinux [ 4.853657] evm: security.ima [ 4.854907] evm: security.capability [ 4.856613] evm: HMAC attrs: 0x1 [ 4.858941] rtc_cmos 00:05: setting system clock to 2026-04-22 14:52:21 UTC (1776869541) [ 4.866905] debug: unmapping init [mem 0xffffffffbe003000-0xffffffffbe1fffff] [ 4.871991] debug: unmapping init [mem 0xffffffffbcd82000-0xffffffffbd058fff] [ 4.881060] Write protecting the kernel read-only data: 28672k [ 4.884536] debug: unmapping init [mem 0xffffffffbb403000-0xffffffffbb5fffff] [ 4.887613] debug: unmapping init [mem 0xffffffffbbd14000-0xffffffffbbdfffff] [ 4.927567] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.936607] systemd[1]: Detected virtualization kvm. [ 4.938600] systemd[1]: Detected architecture x86-64. [ 4.940495] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.964748] systemd[1]: No hostname configured. [ 4.966934] systemd[1]: Set hostname to . [ 4.968700] random: systemd: uninitialized urandom read (16 bytes read) [ 4.971123] systemd[1]: Initializing machine ID from random generator. [ 5.074583] random: ln: uninitialized urandom read (6 bytes read) [ 5.302990] random: systemd: uninitialized urandom read (16 bytes read) [ 5.310581] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.328678] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.337344] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.243151] device-mapper: uevent: version 1.0.3 [ 6.245881] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.384306] random: fast init done [ 7.384820] virtio_net virtio0 ens2: renamed from eth0 [ 7.750587] scsi host0: ata_piix [ 7.923243] scsi host1: ata_piix [ 7.927163] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.932266] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.378327] random: crng init done [ 12.380378] random: 7 urandom warning(s) missed due to ratelimiting [ 12.784990] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.286898] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.189195] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.038270] SELinux: Disabled at runtime. [ 17.117515] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.129634] systemd[1]: Detected virtualization kvm. [ 17.131740] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.553559] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.558595] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.592386] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.597465] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.602443] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.622095] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.631680] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Local Encrypted Volumes. [ 18.961351] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 19.434490] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.270807] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.274174] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.282186] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.310939] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (6s / no limit)[ 25.222252] Key type dns_resolver registered [ 25.771945] NFS: Registering the id_resolver key type [ 25.774108] Key type id_resolver registered [ 25.775434] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg437-server login: [ 56.008443] libcfs: loading out-of-tree module taints kernel. [ 56.026052] Key type ._llcrypt registered [ 56.027823] Key type .llcrypt registered [ 56.077543] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 64.666453] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 65.310687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 65.318954] alg: No test for adler32 (adler32-zlib) [ 66.419593] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 66.819760] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 68.456263] Key type lgssc registered [ 69.151229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 77.625885] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 96.127880] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 101.196890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 101.207639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 102.320655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 102.336026] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 102.378045] Lustre: lustre-MDT0000: new disk, initializing [ 102.412170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 102.420762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 103.983691] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 109.812114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 109.854881] Lustre: 6482:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 109.879677] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 109.883457] Lustre: Skipped 1 previous similar message [ 109.909403] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 109.934156] Lustre: lustre-MDT0001: new disk, initializing [ 109.959534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 109.969921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 109.973546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 111.386316] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.774486] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 117.451747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 117.560667] Lustre: lustre-OST0000: new disk, initializing [ 117.563674] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 117.585237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 119.779458] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 123.896553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 123.901228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 123.917499] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 125.789666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 125.843377] Lustre: lustre-OST0001: new disk, initializing [ 125.845530] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 125.866219] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 127.914258] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 130.548089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 130.554294] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 130.587288] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 134.571402] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 138.360226] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 145.892879] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing check_logdir /tmp/testlogs/ [ 148.206738] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing yml_node [ 150.118517] Lustre: DEBUG MARKER: Client: 2.17.52.53 [ 151.190143] Lustre: DEBUG MARKER: MDS: 2.17.52.53 [ 152.287881] Lustre: DEBUG MARKER: OSS: 2.17.52.53 [ 152.978485] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Apr 22 10:54:48 EDT 2026 [ 160.194509] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 160.850417] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 176.609222] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 176.609415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 176.609988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 176.616165] Lustre: Skipped 2 previous similar messages [ 181.729824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 181.732910] Lustre: Skipped 6 previous similar messages [ 186.850703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 186.854516] Lustre: Skipped 3 previous similar messages [ 187.362539] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 187.452201] Lustre: server umount lustre-MDT0000 complete [ 189.069370] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776869725 with bad export cookie 15842560942730919491 [ 189.070094] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 189.074275] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 191.970140] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 191.971474] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 191.980751] Lustre: Skipped 2 previous similar messages [ 195.276590] Lustre: server umount lustre-MDT0001 complete [ 196.987790] Lustre: server umount lustre-OST0000 complete [ 198.703785] Lustre: server umount lustre-OST0001 complete [ 201.715075] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 205.476140] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 220.662764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 220.780457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 220.797318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 220.839776] Lustre: lustre-MDT0000: new disk, initializing [ 220.887067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.899911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 222.531210] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 227.337443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 227.380499] Lustre: 14946:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 227.491593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 229.103235] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 232.432699] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 233.796627] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 236.595412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.727316] Lustre: lustre-OST0000: new disk, initializing [ 236.729233] Lustre: Skipped 1 previous similar message [ 236.732170] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 236.735033] Lustre: Skipped 2 previous similar messages [ 236.777407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 236.780503] Lustre: Skipped 1 previous similar message [ 238.085413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 238.088963] Lustre: Skipped 1 previous similar message [ 238.092028] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 238.139815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 238.891879] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 242.375543] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 244.093466] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 244.175396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 245.696182] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 245.780521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 248.290665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 248.291607] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 248.299534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 248.302354] Lustre: Skipped 4 previous similar messages [ 252.564334] Lustre: server umount lustre-OST0000 complete [ 258.530193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 258.530600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 258.537364] Lustre: Skipped 1 previous similar message [ 258.541195] LustreError: Skipped 1 previous similar message [ 268.256127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 268.339761] Lustre: server umount lustre-MDT0000 complete [ 268.769509] LustreError: 15739:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 268.776825] LustreError: 15739:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 269.938630] LustreError: 16685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776869806 with bad export cookie 15842560942730921024 [ 269.941152] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 269.945384] LustreError: 16685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 270.083056] Lustre: server umount lustre-MDT0001 complete [ 271.698469] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:56:47 (1776869807) [ 272.586400] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 273.243808] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:56:49 (1776869809) [ 275.986465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 276.185227] LustreError: 18945:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 276.228872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 277.809379] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 281.570284] LustreError: 18946:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 281.678567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 283.577822] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 286.065406] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 287.880274] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 291.217579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 293.868759] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 296.236634] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 306.145656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 306.148270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.152642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 306.162608] Lustre: Skipped 2 previous similar messages [ 306.166445] Lustre: Skipped 4 previous similar messages [ 310.873294] Lustre: server umount lustre-OST0000 complete [ 312.801496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 316.386489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.392905] Lustre: Skipped 1 previous similar message [ 318.645473] Lustre: server umount lustre-MDT0000 complete [ 320.349336] LustreError: 20559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776869856 with bad export cookie 15842560942730921906 [ 320.352957] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.354270] LustreError: 20559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 320.502205] Lustre: server umount lustre-MDT0001 complete [ 324.990958] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 326.236220] Key type lgssc unregistered [ 326.390429] LNet: 22119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.394601] LNetError: 22119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 326.406330] LNet: Removed LNI 192.168.204.137@tcp [ 326.793160] Key type .llcrypt unregistered [ 326.794952] Key type ._llcrypt unregistered [ 337.224794] Key type ._llcrypt registered [ 337.226300] Key type .llcrypt registered [ 337.281288] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 337.776191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 337.788572] alg: No test for adler32 (adler32-zlib) [ 338.687886] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 338.804538] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 340.408487] Key type lgssc registered [ 340.908669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 344.641396] loop: module loaded [ 344.669890] loop0: detected capacity change from 0 to 204800000 [ 344.766597] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 344.775482] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 344.914290] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 347.698613] loop0: detected capacity change from 0 to 204800000 [ 347.788668] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 347.797349] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 347.804141] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 350.028274] loop0: detected capacity change from 0 to 204800000 [ 352.411282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 352.419568] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 353.535776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 353.580317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 353.623327] Lustre: lustre-MDT0000: new disk, initializing [ 353.664488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 353.672366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 355.410056] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 359.827677] loop1: detected capacity change from 0 to 204800000 [ 362.047940] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 362.186964] Lustre: lustre-OST07c6: new disk, initializing [ 362.189725] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 362.192810] Lustre: Skipped 1 previous similar message [ 362.226264] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 362.344750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 362.349570] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 362.376660] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 364.716809] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 367.311526] Lustre: 25258:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 367.315557] Lustre: 25258:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 367.319720] Lustre: 25258:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 367.324787] Lustre: 25258:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 367.328635] Lustre: 25258:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 367.332034] Lustre: 25258:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 383.168451] Lustre: 24257:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 383.177406] Lustre: 24257:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 383.181443] Lustre: 24257:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 383.185512] Lustre: 24257:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 383.189470] Lustre: 24257:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 383.192942] Lustre: 24257:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 387.100447] Lustre: 24258:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 387.113636] Lustre: 24258:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 387.120582] Lustre: 24258:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 387.129724] Lustre: 24258:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 387.136251] Lustre: 24258:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 387.143245] Lustre: 24258:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 387.148021] Lustre: 24258:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 387.153128] Lustre: 24258:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 387.157713] Lustre: 24258:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 387.162030] Lustre: 24258:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 387.166320] Lustre: 24258:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 387.169738] Lustre: 24258:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 390.271466] Lustre: 24258:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 390.276012] Lustre: 24258:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 390.279710] Lustre: 24258:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 390.282973] Lustre: 24258:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 390.287287] Lustre: 24258:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 390.292295] Lustre: 24258:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 390.296030] Lustre: 24258:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 390.300465] Lustre: 24258:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 390.304529] Lustre: 24258:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 390.308206] Lustre: 24258:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 390.312523] Lustre: 24258:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 390.315049] Lustre: 24258:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 397.034482] Lustre: 24256:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 397.039260] Lustre: 24256:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 3 previous similar messages [ 397.042683] Lustre: 24256:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 397.046041] Lustre: 24256:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 397.049286] Lustre: 24256:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 397.053040] Lustre: 24256:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 397.056456] Lustre: 24256:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 397.061450] Lustre: 24256:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 397.065899] Lustre: 24256:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 397.069143] Lustre: 24256:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 397.072350] Lustre: 24256:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 397.074936] Lustre: 24256:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 406.862811] Lustre: 24256:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 406.866257] Lustre: 24256:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 5 previous similar messages [ 406.869454] Lustre: 24256:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 406.872669] Lustre: 24256:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 406.875963] Lustre: 24256:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 406.879116] Lustre: 24256:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 406.882060] Lustre: 24256:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 406.885859] Lustre: 24256:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 406.888890] Lustre: 24256:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 406.891927] Lustre: 24256:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 406.895272] Lustre: 24256:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 406.898286] Lustre: 24256:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 449.505832] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 449.505912] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 449.517204] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 450.147402] Lustre: server umount lustre-OST07c6 complete [ 454.757414] Lustre: server umount lustre-MDT0000 complete [ 461.549481] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 462.845078] Key type lgssc unregistered [ 463.003407] LNet: 27767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 463.006717] LNetError: 27767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 463.016166] LNet: Removed LNI 192.168.204.137@tcp [ 463.382670] Key type .llcrypt unregistered [ 463.384529] Key type ._llcrypt unregistered [ 471.788934] Key type ._llcrypt registered [ 471.790370] Key type .llcrypt registered [ 471.835896] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 472.282181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 472.306794] alg: No test for adler32 (adler32-zlib) [ 473.185164] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 473.291463] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 474.896678] Key type lgssc registered [ 475.338592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 478.449334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 478.459213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.665068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 481.232325] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 484.711778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.862443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 486.309392] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 488.463558] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 490.214851] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 493.092719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 493.263151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.714554] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 498.137634] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 505.581597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 516.066916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 516.067120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.069878] LustreError: Skipped 1 previous similar message [ 516.075786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 516.079171] Lustre: Skipped 1 previous similar message [ 520.537821] Lustre: server umount lustre-OST0000 complete [ 525.793387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 525.797173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.802224] Lustre: Skipped 1 previous similar message [ 525.804544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 528.166571] Lustre: server umount lustre-MDT0000 complete [ 529.801175] LustreError: 31460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870066 with bad export cookie 8127553096696419428 [ 529.803736] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.806432] LustreError: 31460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 529.961549] Lustre: server umount lustre-MDT0001 complete [ 534.410039] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 535.644525] Key type lgssc unregistered [ 535.791858] LNet: 32409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 535.796608] LNetError: 32409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 535.806635] LNet: Removed LNI 192.168.204.137@tcp [ 536.166718] Key type .llcrypt unregistered [ 536.168728] Key type ._llcrypt unregistered [ 545.109762] Key type ._llcrypt registered [ 545.111323] Key type .llcrypt registered [ 545.167986] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 545.710765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 545.720524] alg: No test for adler32 (adler32-zlib) [ 546.621302] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 546.733461] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 548.336470] Key type lgssc registered [ 548.787484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 552.013172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 552.021548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.217340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 554.737970] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 558.109080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.293078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 559.816708] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 562.123313] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 563.532311] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 566.305091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 566.476565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 568.594941] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 570.734483] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 577.770778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 588.257788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 588.259344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.260949] LustreError: Skipped 1 previous similar message [ 588.261647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 588.266427] Lustre: Skipped 1 previous similar message [ 591.122676] Lustre: server umount lustre-OST0000 complete [ 593.377846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.383740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 593.386472] Lustre: Skipped 2 previous similar messages [ 598.497028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 598.499585] Lustre: Skipped 1 previous similar message [ 598.679733] Lustre: server umount lustre-MDT0000 complete [ 600.042154] LustreError: 33622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870136 with bad export cookie 15238540492048707061 [ 600.045611] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 600.047145] LustreError: 33622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 600.179193] Lustre: server umount lustre-MDT0001 complete [ 604.133131] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 605.260559] Key type lgssc unregistered [ 605.402545] LNet: 36837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 605.405695] LNetError: 36837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 605.412462] LNet: Removed LNI 192.168.204.137@tcp [ 605.749292] Key type .llcrypt unregistered [ 605.751137] Key type ._llcrypt unregistered [ 615.599575] Key type ._llcrypt registered [ 615.601031] Key type .llcrypt registered [ 615.644698] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 616.187152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 616.195331] alg: No test for adler32 (adler32-zlib) [ 617.073060] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 617.168183] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 618.768164] Key type lgssc registered [ 619.197256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 622.375931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 622.385650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.620294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 625.232270] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 628.623745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.767181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 630.296672] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 632.449962] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 633.819737] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 636.270168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 636.407090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 638.533686] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 640.570172] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 649.701906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 654.304898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 654.309025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 654.314804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 654.819871] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 654.820751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 654.826375] Lustre: Skipped 1 previous similar message [ 655.912670] Lustre: server umount lustre-MDT0000 complete [ 656.394977] LustreError: 39583:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 656.403555] LustreError: 39583:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 657.417524] LustreError: 38192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870194 with bad export cookie 17792339636756816983 [ 657.422409] LustreError: 38192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 657.423039] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.571539] Lustre: server umount lustre-MDT0001 complete [ 670.596602] LustreError: 39826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.37@tcp arrived at 1776870207 with bad export cookie 17792339636756817634 [ 670.603622] LustreError: 39826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 696.289486] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 696.354740] Lustre: server umount lustre-OST0000 complete [ 702.431688] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 703.546563] Key type lgssc unregistered [ 703.691462] LNet: 41736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.694439] LNetError: 41736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.707571] LNet: Removed LNI 192.168.204.137@tcp [ 704.036810] Key type .llcrypt unregistered [ 704.038601] Key type ._llcrypt unregistered [ 713.666763] Key type ._llcrypt registered [ 713.668499] Key type .llcrypt registered [ 713.716239] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 714.124992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 714.193234] alg: No test for adler32 (adler32-zlib) [ 715.076114] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 715.177456] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 716.768214] Key type lgssc registered [ 717.178606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 720.037471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 720.046695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.211753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 722.549427] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 725.730989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 725.867530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 727.285858] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 729.265930] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 730.520581] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 732.867463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 733.003103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 735.044393] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 736.921407] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 746.346092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 751.587949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 751.588375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 751.593140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 751.595609] LustreError: Skipped 1 previous similar message [ 752.646066] Lustre: lustre-OST0000: Not available for connect from 192.168.204.37@tcp (stopping) [ 752.649016] Lustre: Skipped 1 previous similar message [ 756.705811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 756.708399] Lustre: Skipped 1 previous similar message [ 761.829854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 761.833200] Lustre: Skipped 2 previous similar messages [ 763.873414] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 763.930798] Lustre: server umount lustre-OST0000 complete [ 776.821139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 776.821834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 776.826628] Lustre: Skipped 2 previous similar messages [ 776.828864] Lustre: Skipped 1 previous similar message [ 782.995524] Lustre: server umount lustre-MDT0000 complete [ 784.286269] LustreError: 43095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870320 with bad export cookie 3435496879479740334 [ 784.288106] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 784.289908] LustreError: 43095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 790.442414] Lustre: server umount lustre-MDT0001 complete [ 794.014498] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 795.040522] Key type lgssc unregistered [ 795.163362] LNet: 46482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.166491] LNetError: 46482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.176306] LNet: Removed LNI 192.168.204.137@tcp [ 795.474347] Key type .llcrypt unregistered [ 795.476017] Key type ._llcrypt unregistered [ 804.198658] Key type ._llcrypt registered [ 804.199889] Key type .llcrypt registered [ 804.232374] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 804.560444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 804.594442] alg: No test for adler32 (adler32-zlib) [ 805.453867] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 805.539805] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 807.120138] Key type lgssc registered [ 807.436266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 809.796542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 809.802814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.956540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 812.191321] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 814.934128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 815.061047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 816.419642] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 818.179521] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 819.392546] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 821.624844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 821.735863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 823.568663] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 825.371551] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 836.074962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 838.189873] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 839.290669] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 849.985255] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-2211h06m25s [ 850.488669] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 851.427283] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 851.427729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 851.431065] Lustre: Skipped 1 previous similar message [ 852.449692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 852.451934] Lustre: Skipped 5 previous similar messages [ 857.303119] Lustre: server umount lustre-MDT0000 complete [ 858.486292] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 858.594592] LustreError: 47858:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 858.601069] LustreError: 47858:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 860.695287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.737963] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 860.818638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 860.850642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 861.866543] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 862.120727] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 863.233531] Lustre: DEBUG MARKER: Wait for df (36005) ... [ 863.740183] Lustre: DEBUG MARKER: done [ 865.760777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 865.763441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 865.767383] Lustre: Skipped 1 previous similar message [ 865.769411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 865.771217] Lustre: Skipped 1 previous similar message [ 866.273113] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 870.798097] Lustre: server umount lustre-OST0000 complete [ 872.417434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 872.417484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 872.417938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 872.417942] Lustre: Skipped 3 previous similar messages [ 872.421693] Lustre: Skipped 2 previous similar messages [ 878.231334] Lustre: server umount lustre-MDT0000 complete [ 878.497147] LustreError: 50225:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 878.503652] LustreError: 50225:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 879.476305] LustreError: 49473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870416 with bad export cookie 3931155410652737875 [ 879.480240] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 879.480617] LustreError: 49473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 879.604035] Lustre: server umount lustre-MDT0001 complete [ 883.027514] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 883.942960] Key type lgssc unregistered [ 884.076396] LNet: 52420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.079358] LNetError: 52420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.086394] LNet: Removed LNI 192.168.204.137@tcp [ 884.367976] Key type .llcrypt unregistered [ 884.369506] Key type ._llcrypt unregistered [ 897.837391] Key type ._llcrypt registered [ 897.838643] Key type .llcrypt registered [ 897.880833] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 898.174720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 898.186691] alg: No test for adler32 (adler32-zlib) [ 899.052680] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 899.144616] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 900.728167] Key type lgssc registered [ 901.096766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 903.701795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 903.708129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 904.801759] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 904.811159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 904.874509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 906.163639] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 909.030494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 909.073256] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 909.084289] Lustre: 54590:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 909.181584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 910.545624] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 912.378691] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 913.610048] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 915.874641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 915.936398] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 915.996578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 917.780806] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 919.524498] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 924.168550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 927.847401] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-2211h07m43s [ 928.393825] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 930.704366] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 931.537303] Lustre: DEBUG MARKER: Creating a test file and stat it [ 931.554784] LustreError: 54597:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 951.819955] Lustre: lustre-MDT0000: Client ca5b6ccb-eb9d-47ec-8ce6-33a4ab972417 (at 192.168.204.37@tcp) reconnecting [ 971.600100] LustreError: 54597:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 972.268336] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 973.000242] Lustre: DEBUG MARKER: done [ 975.329426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 975.331234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.332363] LustreError: Skipped 1 previous similar message [ 975.332904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 975.335950] Lustre: Skipped 1 previous similar message [ 975.341257] Lustre: Skipped 1 previous similar message [ 980.450580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 980.452712] Lustre: Skipped 1 previous similar message [ 980.621587] Lustre: server umount lustre-OST0000 complete [ 985.569418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 985.569876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 985.572759] Lustre: Skipped 1 previous similar message [ 991.649072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 991.651443] Lustre: Skipped 4 previous similar messages [ 996.320127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 996.385815] Lustre: server umount lustre-MDT0000 complete [ 997.598031] LustreError: 54582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870534 with bad export cookie 459621148139306768 [ 997.601054] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 997.602569] LustreError: 54582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 997.713985] Lustre: server umount lustre-MDT0001 complete [ 1001.219676] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 1002.207112] Key type lgssc unregistered [ 1002.341410] LNet: 58558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.344319] LNetError: 58558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.356371] LNet: Removed LNI 192.168.204.137@tcp [ 1002.649307] Key type .llcrypt unregistered [ 1002.650422] Key type ._llcrypt unregistered [ 1017.755088] Key type ._llcrypt registered [ 1017.756299] Key type .llcrypt registered [ 1017.791312] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 1018.203311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1018.214737] alg: No test for adler32 (adler32-zlib) [ 1019.071389] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 1019.149348] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 1020.728158] Key type lgssc registered [ 1021.061516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1022.040885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1022.046524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.133224] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1023.141490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1023.196920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1023.520349] Lustre: Failing over lustre-MDT0000 [ 1023.612163] Lustre: server umount lustre-MDT0000 complete [ 1025.504773] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:09:21 (1776870561) [ 1027.664021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1027.819150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1029.066159] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1031.901859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.937210] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1031.948619] Lustre: 61316:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1032.043875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1033.354318] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1035.177761] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1036.409087] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1038.544520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1038.611122] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1038.668527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1040.469131] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1042.203982] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1045.003818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1050.408893] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1052.640787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1052.643450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1052.647751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1055.202216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1055.202839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.204536] Lustre: Skipped 1 previous similar message [ 1055.208855] Lustre: Skipped 1 previous similar message [ 1058.136231] Lustre: server umount lustre-MDT0000 complete [ 1059.445327] LustreError: 61308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870596 with bad export cookie 8260066775477768300 [ 1059.448407] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1059.450148] LustreError: 61308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1059.551650] Lustre: server umount lustre-MDT0001 complete [ 1060.611118] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1062.071290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1064.736587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.536616] LustreError: 60239:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000599a2324 x1863183819111680/t0(0) o250->MGC192.168.204.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1069.671248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.697053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1070.886908] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1073.679090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.772259] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.852902] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1075.016884] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1075.872101] Lustre: 60241:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776870596/real 1776870596] req@00000000377c12c2 x1863183819111296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776870612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1076.723175] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1077.914225] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1080.289636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1080.290962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1080.293648] Lustre: Skipped 1 previous similar message [ 1080.295374] Lustre: Skipped 1 previous similar message [ 1081.312115] Lustre: 60242:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776870600/real 1776870600] req@00000000843235c7 x1863183819111424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776870616 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1082.336138] Lustre: 60243:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776870602/real 1776870602] req@000000006eb905d1 x1863183819111552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776870618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1085.408124] Lustre: 60240:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776870606/real 1776870606] req@000000000d806b00 x1863183819111808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776870622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1085.409235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1085.419191] Lustre: Skipped 4 previous similar messages [ 1085.583683] Lustre: server umount lustre-MDT0000 complete [ 1086.713233] LustreError: 61307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870623 with bad export cookie 8260066775477770568 [ 1086.716048] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1086.717604] LustreError: 61307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1086.823342] Lustre: server umount lustre-MDT0001 complete [ 1088.763479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1091.694695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.672484] LustreError: 60239:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000015ca2958 x1863183819130752/t0(0) o250->MGC192.168.204.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1096.776520] LustreError: 62934:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1096.783174] LustreError: 62934:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1096.798241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1096.800368] Lustre: Skipped 1 previous similar message [ 1096.820600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1098.047717] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1100.813071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1100.905128] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1100.910623] Lustre: Skipped 1 previous similar message [ 1102.173411] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1103.827444] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1104.938714] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1106.404555] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1106.406874] Lustre: Skipped 1 previous similar message [ 1107.424166] Lustre: 60240:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776870627/real 1776870627] req@0000000052f18344 x1863183819130368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776870643 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1110.501844] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1115.618173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.620334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1115.622814] Lustre: Skipped 2 previous similar messages [ 1115.624651] Lustre: Skipped 1 previous similar message [ 1118.099503] Lustre: server umount lustre-MDT0000 complete [ 1119.270114] LustreError: 63548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870655 with bad export cookie 8260066775477771947 [ 1119.273065] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1119.273794] LustreError: 63548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1119.366761] Lustre: server umount lustre-MDT0001 complete [ 1121.437257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1122.465520] Lustre: DEBUG MARKER: files compared the same [ 1133.405776] Lustre: server umount lustre-OST0000 complete [ 1138.906705] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 1139.999529] Key type lgssc unregistered [ 1140.135404] LNet: 70221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.137817] LNetError: 70221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1140.145318] LNet: Removed LNI 192.168.204.137@tcp [ 1140.471238] Key type .llcrypt unregistered [ 1140.472639] Key type ._llcrypt unregistered [ 1148.954243] Key type ._llcrypt registered [ 1148.955333] Key type .llcrypt registered [ 1148.987829] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 1149.398767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1149.406439] alg: No test for adler32 (adler32-zlib) [ 1150.249689] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 1150.318960] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 1151.896147] Key type lgssc registered [ 1152.229678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1154.593229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1154.599224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1155.741431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1156.912959] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug +malloc all [ 1159.485597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1159.574842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1160.857838] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug +malloc all [ 1162.582803] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1163.730366] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1165.970972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1166.066252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1167.922588] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug +malloc all [ 1169.697976] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1180.387605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1190.881274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.881439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1190.881647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1190.887170] Lustre: Skipped 1 previous similar message [ 1190.895785] LustreError: Skipped 1 previous similar message [ 1193.866430] Lustre: server umount lustre-OST0000 complete [ 1195.488570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1195.490747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1195.494521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1195.496155] Lustre: Skipped 1 previous similar message [ 1201.120631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1201.123629] Lustre: Skipped 2 previous similar messages [ 1201.244339] Lustre: server umount lustre-MDT0000 complete [ 1202.477839] LustreError: 71585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870739 with bad export cookie 12613298203004387033 [ 1202.479294] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1202.483338] LustreError: 71585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1202.586829] Lustre: server umount lustre-MDT0001 complete [ 1205.977437] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 1206.923167] Key type lgssc unregistered [ 1207.049390] LNet: 74800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1207.051958] LNetError: 74800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1207.060321] LNet: Removed LNI 192.168.204.137@tcp [ 1207.335214] Key type .llcrypt unregistered [ 1207.336599] Key type ._llcrypt unregistered [ 1216.078724] Key type ._llcrypt registered [ 1216.079774] Key type .llcrypt registered [ 1216.121216] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 1216.544622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1216.552233] alg: No test for adler32 (adler32-zlib) [ 1217.406448] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 1217.487051] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 1219.064099] Key type lgssc registered [ 1219.416142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1221.944978] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1221.950459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1237.408234] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1242.528233] LustreError: 76150:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.137@tcp: failed processing log, type 4: rc = -110 [ 1269.152168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.007333] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1272.716667] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1275.017308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.062095] Lustre: *** cfs_fail_loc=706, val=0*** [ 1275.180636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1275.201117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1276.348407] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1279.013224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1279.128803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1280.337727] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1282.016930] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1283.112876] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1284.578933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.579299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1284.583370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1284.587606] Lustre: Skipped 1 previous similar message [ 1285.838684] Lustre: server umount lustre-OST0000 complete [ 1289.698053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1289.698472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.706368] Lustre: Skipped 1 previous similar message [ 1289.709202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1289.711858] Lustre: Skipped 1 previous similar message [ 1293.212700] Lustre: server umount lustre-MDT0000 complete [ 1294.402635] LustreError: 78670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870831 with bad export cookie 16453259992456031482 [ 1294.404990] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1294.408086] LustreError: 78670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1294.511364] Lustre: server umount lustre-MDT0001 complete [ 1297.676312] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 1298.676534] Key type lgssc unregistered [ 1298.804339] LNet: 79403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.808138] LNetError: 79403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.814310] LNet: Removed LNI 192.168.204.137@tcp [ 1299.098950] Key type .llcrypt unregistered [ 1299.100137] Key type ._llcrypt unregistered [ 1307.824470] Key type ._llcrypt registered [ 1307.825670] Key type .llcrypt registered [ 1307.860142] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 1308.293947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1308.301804] alg: No test for adler32 (adler32-zlib) [ 1309.151846] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 1309.225605] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 1310.808158] Key type lgssc registered [ 1311.183767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1313.756342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1313.762237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1315.995416] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1318.536664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1318.745194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1320.086382] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1322.951553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1323.085816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1325.018973] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1327.773611] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1327.779649] Lustre: MGS: Client a8b50441-f661-4e9c-a787-0e1d59b3c7c3 (at 0@lo) reconnecting [ 1327.865435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1327.904717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1329.182761] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1338.337220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1338.341162] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1338.347812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1339.363459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1339.363679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1339.364399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1343.055145] Lustre: server umount lustre-OST0000 complete [ 1344.481561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1344.481564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1344.482446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1344.489168] Lustre: Skipped 2 previous similar messages [ 1349.600983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1349.602868] Lustre: Skipped 1 previous similar message [ 1350.410699] Lustre: server umount lustre-MDT0000 complete [ 1351.757433] Lustre: server umount lustre-MDT0001 complete [ 1352.961453] Lustre: server umount MGS complete [ 1356.662238] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:14:52 (1776870892) [ 1365.727945] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 1369.022447] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 1380.870093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1382.167348] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1385.987850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.072580] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1386.087918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1386.122402] Lustre: lustre-MDT0001: new disk, initializing [ 1386.145908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1386.155167] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1386.159281] Lustre: 88504:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1386.164049] LustreError: 88504:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1387.412815] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1391.563738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1391.656623] Lustre: lustre-OST0000: new disk, initializing [ 1391.658849] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1391.679265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1393.505476] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1396.633090] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1398.816653] Lustre: 89979:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000005cab798f x1863184123200512/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1776870951 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1399.014054] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1399.019177] Lustre: MGS: Client 1fba49ae-96c3-4106-a70a-6f13c7cb81e2 (at 0@lo) reconnecting [ 1399.036343] Lustre: 90184:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1399.044893] LustreError: 88488:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1399.049381] LustreError: 88488:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1399.053300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1399.091956] Lustre: lustre-MDT0000: new disk, initializing [ 1399.146686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1400.161889] Lustre: 90211:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000af8891ea x1863184123205888/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1776870952 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1400.502919] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1400.864652] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1407.907118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1407.911396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1409.251259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1409.254202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1409.316487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1429.985222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1429.989323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.992472] Lustre: Skipped 1 previous similar message [ 1429.993957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1435.660667] Lustre: server umount lustre-OST0000 complete [ 1441.249733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1441.249845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1441.250524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1441.250529] Lustre: Skipped 2 previous similar messages [ 1441.259347] Lustre: Skipped 2 previous similar messages [ 1443.209545] Lustre: server umount lustre-MDT0000 complete [ 1450.663587] Lustre: server umount lustre-MDT0001 complete [ 1451.935776] Lustre: server umount MGS complete [ 1455.668211] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:16:31 (1776870991) [ 1461.778173] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 1462.785657] Key type lgssc unregistered [ 1462.932348] LNet: 92674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1462.934825] LNetError: 92674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1462.942272] LNet: Removed LNI 192.168.204.137@tcp [ 1463.225470] Key type .llcrypt unregistered [ 1463.226576] Key type ._llcrypt unregistered [ 1470.124052] Key type ._llcrypt registered [ 1470.125417] Key type .llcrypt registered [ 1470.163409] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 1470.463453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1470.470247] alg: No test for adler32 (adler32-zlib) [ 1471.320486] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 1471.395421] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 1472.976111] Key type lgssc registered [ 1473.314591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1475.909152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1475.918714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1477.088838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1478.407224] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1481.194176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1481.305239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1482.549211] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1484.388574] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1485.634291] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1487.869312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1487.977444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1489.849741] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1491.656626] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1502.310651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1511.905432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1511.908022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.912458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1512.930438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1512.931805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1512.931831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1515.921112] Lustre: server umount lustre-OST0000 complete [ 1518.051080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1518.051762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1518.055018] Lustre: Skipped 1 previous similar message [ 1518.058317] Lustre: Skipped 2 previous similar messages [ 1523.169179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1523.171952] Lustre: Skipped 1 previous similar message [ 1523.284466] Lustre: server umount lustre-MDT0000 complete [ 1524.570201] LustreError: 93836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871061 with bad export cookie 17717363406787686737 [ 1524.573679] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1524.574500] LustreError: 93836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1524.703530] Lustre: server umount lustre-MDT0001 complete [ 1528.435318] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 1529.362913] Key type lgssc unregistered [ 1529.506302] LNet: 97047:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1529.508910] LNetError: 97047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1529.516288] LNet: Removed LNI 192.168.204.137@tcp [ 1529.791550] Key type .llcrypt unregistered [ 1529.793232] Key type ._llcrypt unregistered [ 1536.224498] Key type ._llcrypt registered [ 1536.225621] Key type .llcrypt registered [ 1536.260936] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 1536.621376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1536.655073] alg: No test for adler32 (adler32-zlib) [ 1537.494758] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 1537.575389] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 1539.152124] Key type lgssc registered [ 1539.474533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1541.420743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1541.423109] LustreError: 98091:0:(tgt_mount.c:2520:server_fill_super()) cfs_race id 716 sleeping [ 1541.423410] Lustre: Skipped 1 previous similar message [ 1541.426926] LustreError: 98090:0:(tgt_mount.c:2520:server_fill_super()) cfs_fail_race id 716 waking [ 1541.429089] LustreError: 98091:0:(tgt_mount.c:2520:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1541.431588] LustreError: 98091:0:(tgt_mount.c:2532:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1541.434547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1541.435616] LustreError: 98091:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1542.582873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1545.035096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1545.131466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1546.374583] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1547.972516] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing lsmod [ 1548.925190] LustreError: 99259:0:(tgt_mount.c:2520:server_fill_super()) cfs_race id 716 sleeping [ 1548.933243] LustreError: 99262:0:(tgt_mount.c:2520:server_fill_super()) cfs_fail_race id 716 waking [ 1548.935511] LustreError: 99259:0:(tgt_mount.c:2520:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 1548.937397] LustreError: 99259:0:(tgt_mount.c:2532:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1548.940213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1548.941075] LustreError: 99259:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1549.066922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1550.304789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1550.307424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1550.311562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1554.402129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1554.402853] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1554.405571] Lustre: Skipped 1 previous similar message [ 1554.410256] Lustre: Skipped 1 previous similar message [ 1556.119734] Lustre: server umount lustre-MDT0000 complete [ 1557.301379] LustreError: 98104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871093 with bad export cookie 12151026294606673793 [ 1557.304993] LustreError: 98104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1557.305408] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1558.496245] LustreError: 98558:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ba066198 x1863184363305344/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1558.545278] Lustre: server umount lustre-MDT0001 complete [ 1570.009548] Lustre: server umount lustre-OST0000 complete [ 1572.749450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.906515] LustreError: 100324:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1572.914622] LustreError: 100324:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1572.930200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.189562] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1577.094046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1577.222867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1578.553868] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1580.417550] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1581.714592] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1584.071582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1586.086197] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1587.878587] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1597.543345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1603.042059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1603.044178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1603.044238] LustreError: Skipped 1 previous similar message [ 1603.047977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1603.051627] Lustre: Skipped 1 previous similar message [ 1605.904306] Lustre: server umount lustre-OST0000 complete [ 1608.162243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1608.162289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.168472] Lustre: Skipped 1 previous similar message [ 1608.171103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1608.172733] Lustre: Skipped 1 previous similar message [ 1613.281357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1613.283215] Lustre: Skipped 1 previous similar message [ 1613.341924] Lustre: server umount lustre-MDT0000 complete [ 1614.583027] LustreError: 100306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871151 with bad export cookie 12151026294606674451 [ 1614.585808] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1614.586665] LustreError: 100306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1614.699561] Lustre: server umount lustre-MDT0001 complete [ 1618.063075] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 1619.046212] Key type lgssc unregistered [ 1619.173339] LNet: 103519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1619.175793] LNetError: 103519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1619.184283] LNet: Removed LNI 192.168.204.137@tcp [ 1619.462591] Key type .llcrypt unregistered [ 1619.463917] Key type ._llcrypt unregistered [ 1627.988152] Key type ._llcrypt registered [ 1627.989191] Key type .llcrypt registered [ 1628.023871] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 1628.348889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1628.380464] alg: No test for adler32 (adler32-zlib) [ 1629.243126] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 1629.326301] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 1630.904157] Key type lgssc registered [ 1631.236694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1633.749653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1633.756855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1634.914077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1636.222910] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1638.988203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1639.112150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1640.366575] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1642.144266] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1643.290083] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1645.503131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1645.539705] LustreError: 106486:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1645.544785] LustreError: 106486:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1645.615281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1647.455444] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1648.681830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1649.232703] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1652.114978] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1652.861446] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1658.854037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1658.854321] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1658.855298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1658.861524] Lustre: Skipped 1 previous similar message [ 1658.866641] LustreError: Skipped 1 previous similar message [ 1663.970113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1663.972821] Lustre: Skipped 2 previous similar messages [ 1668.064092] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1668.120360] Lustre: server umount lustre-OST0000 complete [ 1670.112950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1670.115238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1670.118850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1674.209168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1674.209579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1675.478652] Lustre: server umount lustre-MDT0000 complete [ 1676.647110] LustreError: 104876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871213 with bad export cookie 1888585089649563212 [ 1676.650544] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1676.652531] LustreError: 104876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1676.765926] Lustre: server umount lustre-MDT0001 complete [ 1680.169660] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 1681.157554] Key type lgssc unregistered [ 1681.287461] LNet: 108172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1681.290333] LNetError: 108172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1681.305300] LNet: Removed LNI 192.168.204.137@tcp [ 1681.596650] Key type .llcrypt unregistered [ 1681.598038] Key type ._llcrypt unregistered [ 1688.308582] Key type ._llcrypt registered [ 1688.309804] Key type .llcrypt registered [ 1688.346869] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 1688.701291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1688.747087] alg: No test for adler32 (adler32-zlib) [ 1689.601339] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 1689.688121] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 1691.264113] Key type lgssc registered [ 1691.607960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1694.165096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1694.171100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1695.322507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1696.599725] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1699.383334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.499148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1700.854406] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1702.717962] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1703.858343] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1706.053678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1706.160635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1707.954880] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1709.646479] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1719.464086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1724.899371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1724.900227] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1724.900771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1724.903485] Lustre: Skipped 1 previous similar message [ 1725.011944] Lustre: server umount lustre-OST0000 complete [ 1728.407950] Lustre: server umount lustre-MDT0000 complete [ 1729.625501] LustreError: 110071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871266 with bad export cookie 12991552922511825187 [ 1729.629342] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1729.631331] LustreError: 110071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1729.755127] Lustre: server umount lustre-MDT0001 complete [ 1733.148745] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 1734.112258] Key type lgssc unregistered [ 1734.247362] LNet: 112544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1734.249608] LNetError: 112544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1734.260332] LNet: Removed LNI 192.168.204.137@tcp [ 1734.523736] Key type .llcrypt unregistered [ 1734.524789] Key type ._llcrypt unregistered [ 1743.146481] Key type ._llcrypt registered [ 1743.147629] Key type .llcrypt registered [ 1743.184567] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 1743.551904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1743.580074] alg: No test for adler32 (adler32-zlib) [ 1744.431472] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 1744.505528] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 1746.088104] Key type lgssc registered [ 1746.446074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1749.031663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1749.038688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1750.199575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1751.512203] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1754.272451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1754.389352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1755.675815] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1757.510384] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1758.727490] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1760.926014] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1761.038522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.900880] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1764.600461] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1774.375837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1777.531968] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1778.690455] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1778.692976] Lustre: Skipped 2 previous similar messages [ 1780.060075] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1780.062416] Lustre: Skipped 2 previous similar messages [ 1787.899391] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1787.910034] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1790.324651] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.37@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1790.329183] Lustre: Skipped 2 previous similar messages [ 1793.656531] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.37@tcp 192.168.0.[2,10]@tcp [ 1797.644274] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.37/24@tcp in log lustre-MDT0000 [ 1797.646859] Lustre: Skipped 2 previous similar messages [ 1801.657429] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.37/24@tcp [ 1801.659746] Lustre: Skipped 1 previous similar message [ 1805.282931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1805.283432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1805.284262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1805.290307] Lustre: Skipped 1 previous similar message [ 1810.058330] Lustre: server umount lustre-OST0000 complete [ 1815.521479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1815.522325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1815.524706] Lustre: Skipped 1 previous similar message [ 1815.526383] Lustre: Skipped 1 previous similar message [ 1817.571907] Lustre: server umount lustre-MDT0000 complete [ 1818.886323] LustreError: 113948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871355 with bad export cookie 271470484295276307 [ 1818.889462] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1818.889955] LustreError: 113948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1819.018547] Lustre: server umount lustre-MDT0001 complete [ 1822.487486] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 1823.531812] Key type lgssc unregistered [ 1823.650369] LNet: 118703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1823.652977] LNetError: 118703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.660271] LNet: Removed LNI 192.168.204.137@tcp [ 1823.946799] Key type .llcrypt unregistered [ 1823.948040] Key type ._llcrypt unregistered [ 1834.383579] Key type ._llcrypt registered [ 1834.384833] Key type .llcrypt registered [ 1834.434761] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 1834.924882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1834.956947] alg: No test for adler32 (adler32-zlib) [ 1835.825742] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 1835.922286] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 1837.512184] Key type lgssc registered [ 1837.895180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1840.549346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1840.555642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.707661] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1841.713695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1841.717561] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.37/24@tcp [ 1843.029899] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1845.787802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1845.891471] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1845.899451] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.37/24@tcp [ 1845.914211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1847.401962] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1849.246190] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1850.538551] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1852.867066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1852.996669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.908411] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1856.642655] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1867.368596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 1876.961872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1876.964588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1876.969050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1877.986159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1877.987393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.988239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1881.494549] Lustre: server umount lustre-OST0000 complete [ 1883.106025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1883.106495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1883.110037] Lustre: Skipped 1 previous similar message [ 1883.113270] Lustre: Skipped 2 previous similar messages [ 1888.225370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1888.227203] Lustre: Skipped 1 previous similar message [ 1888.924707] Lustre: server umount lustre-MDT0000 complete [ 1890.214184] LustreError: 121845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871426 with bad export cookie 10196465801778753719 [ 1890.216719] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1890.217828] LustreError: 121845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1890.340659] Lustre: server umount lustre-MDT0001 complete [ 1893.879797] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 1894.927192] Key type lgssc unregistered [ 1895.066629] LNet: 123475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.070276] LNetError: 123475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1895.077409] LNet: Removed LNI 192.168.204.137@tcp [ 1895.396396] Key type .llcrypt unregistered [ 1895.398844] Key type ._llcrypt unregistered [ 1904.652562] Key type ._llcrypt registered [ 1904.653660] Key type .llcrypt registered [ 1904.696213] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 1910.725195] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 1911.077028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1911.084529] alg: No test for adler32 (adler32-zlib) [ 1911.943517] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 1912.034539] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 1913.616145] Key type lgssc registered [ 1914.023432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1925.318499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1925.326058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1926.422439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1926.433158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1926.469978] Lustre: lustre-MDT0000: new disk, initializing [ 1926.493609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1926.500121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1927.763981] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1931.792309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.833155] Lustre: 126815:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1931.855562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1931.857962] Lustre: Skipped 1 previous similar message [ 1931.896284] Lustre: lustre-MDT0001: new disk, initializing [ 1931.926588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1931.937686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1931.942346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1933.281531] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1936.382800] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1937.546806] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1939.755659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1939.861252] Lustre: lustre-OST0000: new disk, initializing [ 1939.864669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1939.896960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.131814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1941.134924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1941.175758] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1941.915125] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1945.008667] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1946.342301] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1946.412063] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1947.878913] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1947.969418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1951.714678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1951.715389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.716825] LustreError: Skipped 1 previous similar message [ 1951.717335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1951.721955] Lustre: Skipped 1 previous similar message [ 1954.775078] Lustre: server umount lustre-OST0000 complete [ 1956.834118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1956.835012] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1956.838653] Lustre: Skipped 1 previous similar message [ 1956.840482] Lustre: Skipped 1 previous similar message [ 1961.953283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1961.955385] Lustre: Skipped 2 previous similar messages [ 1962.208729] Lustre: server umount lustre-MDT0000 complete [ 1963.487088] LustreError: 126807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871500 with bad export cookie 6768628656310958849 [ 1963.489581] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1963.490724] LustreError: 126807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1963.605715] Lustre: server umount lustre-MDT0001 complete [ 1966.567870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1966.757808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.107267] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1970.869512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1970.991663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1972.295057] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1974.205616] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1975.450473] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1977.652595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1979.664425] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1981.517771] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1982.839649] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1993.198794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 1994.575606] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1994.653404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1998.844135] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1998.922405] Lustre: lustre-OST0001: new disk, initializing [ 1998.925845] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1998.957092] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1998.959596] Lustre: Skipped 1 previous similar message [ 2001.181366] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2004.768671] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2005.504863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2005.507780] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2005.539613] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2005.864464] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2007.124427] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2007.198232] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2008.369478] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff917d9047b800.ost_server_uuid 50 [ 2008.921298] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff917d9047b800.ost_server_uuid in FULL state after 0 sec [ 2015.713640] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2015.714287] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.717276] LustreError: Skipped 1 previous similar message [ 2015.723690] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2015.726786] Lustre: Skipped 1 previous similar message [ 2016.957971] Lustre: server umount lustre-OST0001 complete [ 2020.835295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.836273] LustreError: 132220:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2020.836768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2020.840206] Lustre: Skipped 2 previous similar messages [ 2020.844686] LustreError: 132220:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2024.265640] Lustre: server umount lustre-OST0000 complete [ 2025.953507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.960407] Lustre: Skipped 1 previous similar message [ 2031.073973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2031.078427] Lustre: Skipped 4 previous similar messages [ 2031.590968] Lustre: server umount lustre-MDT0000 complete [ 2033.062422] LustreError: 130607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871569 with bad export cookie 6768628656310959717 [ 2033.066388] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2033.066492] LustreError: 130607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2033.209957] Lustre: server umount lustre-MDT0001 complete [ 2040.958320] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 2042.163316] Key type lgssc unregistered [ 2042.305636] LNet: 136530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2042.309585] LNetError: 136530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2042.321583] LNet: Removed LNI 192.168.204.137@tcp [ 2042.722847] Key type .llcrypt unregistered [ 2042.725056] Key type ._llcrypt unregistered [ 2056.947494] Key type ._llcrypt registered [ 2056.948660] Key type .llcrypt registered [ 2056.990628] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 2063.236098] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 2063.671195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2063.693103] alg: No test for adler32 (adler32-zlib) [ 2064.538704] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 2064.617411] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 2066.192134] Key type lgssc registered [ 2066.585363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2078.256519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2078.262053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.357115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2079.372123] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2079.415127] Lustre: lustre-MDT0000: new disk, initializing [ 2079.459104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.466848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2080.879393] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2085.154860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2085.208887] Lustre: 140621:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2085.232087] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2085.234645] Lustre: Skipped 1 previous similar message [ 2085.276556] Lustre: lustre-MDT0001: new disk, initializing [ 2085.314275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2085.329329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2085.333552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2086.664267] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2089.789782] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2091.033107] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2093.346368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2093.452897] Lustre: lustre-OST0000: new disk, initializing [ 2093.456212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2093.492249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2095.413752] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2098.825708] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2099.727579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2099.734983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2099.784542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2106.134635] Lustre: Failing over lustre-OST0000 [ 2106.194646] Lustre: server umount lustre-OST0000 complete [ 2119.164272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2119.276053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.279458] Lustre: Skipped 1 previous similar message [ 2119.282923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2119.290254] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2120.721837] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2121.409204] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2123.181368] Lustre: Failing over lustre-MDT0000 [ 2123.297727] Lustre: server umount lustre-MDT0000 complete [ 2124.772635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.782335] Lustre: Skipped 2 previous similar messages [ 2125.792192] Lustre: 138850:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776871646/real 1776871646] req@00000000359b73b5 x1863184915918336/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776871662 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2125.805584] Lustre: 138850:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2129.890113] LustreError: 140633:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2129.897665] LustreError: 140633:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 2130.912262] Lustre: 138850:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776871651/real 1776871651] req@000000005cdc1d13 x1863184915920256/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776871667 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2130.942750] LustreError: 141957:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2135.009265] LustreError: 140632:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2135.013908] LustreError: 140632:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2136.850131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.900653] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2136.906522] Lustre: 140624:0:(mgc_request.c:1897:mgc_process_log()) MGC192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2136.982613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2137.001379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2137.023835] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2138.340683] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2141.182389] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2142.186679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2142.197859] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2142.227923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2147.300157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2147.301663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.302788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2147.305794] Lustre: Skipped 1 previous similar message [ 2149.531976] Lustre: server umount lustre-OST0000 complete [ 2152.420917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2152.422352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.423779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2152.423788] Lustre: Skipped 1 previous similar message [ 2152.427823] LustreError: Skipped 1 previous similar message [ 2152.446128] Lustre: Skipped 1 previous similar message [ 2157.537961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2157.540345] Lustre: Skipped 2 previous similar messages [ 2162.658364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2162.662036] Lustre: Skipped 1 previous similar message [ 2165.216153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2165.315512] Lustre: server umount lustre-MDT0000 complete [ 2166.911570] LustreError: 142361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871703 with bad export cookie 7827486382587418504 [ 2166.916214] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2166.916532] LustreError: 142361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2167.071555] Lustre: server umount lustre-MDT0001 complete [ 2171.220729] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 2172.312209] Key type lgssc unregistered [ 2172.451440] LNet: 145856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.454744] LNetError: 145856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2172.468322] LNet: Removed LNI 192.168.204.137@tcp [ 2172.899957] Key type .llcrypt unregistered [ 2172.902257] Key type ._llcrypt unregistered [ 2192.938255] Key type ._llcrypt registered [ 2192.939348] Key type .llcrypt registered [ 2192.976476] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 2193.365781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2193.435833] alg: No test for adler32 (adler32-zlib) [ 2194.307087] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 2194.404090] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 2195.992316] Key type lgssc registered [ 2196.434195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2199.160263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2199.167302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2200.339408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.546102] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2204.568655] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2206.539091] LustreError: 147385:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2206.544541] LustreError: 147385:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2206.549071] LustreError: 147385:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2206.658204] Lustre: server umount lustre-MDT0000 complete [ 2211.295752] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 2214.226198] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 2225.629223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2225.719730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2225.730376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2225.765688] Lustre: lustre-MDT0000: new disk, initializing [ 2225.788401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2225.794327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2227.070780] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2231.070718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2231.107675] Lustre: 150536:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2231.128686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2231.130828] Lustre: Skipped 1 previous similar message [ 2231.167623] Lustre: lustre-MDT0001: new disk, initializing [ 2231.199029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2231.211096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2231.215896] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2232.474777] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2235.593034] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2236.762435] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2239.171831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2239.289624] Lustre: lustre-OST0000: new disk, initializing [ 2239.293052] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2239.327950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.383349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2239.386652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2239.442610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2241.299487] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2244.677969] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2246.522112] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2247.102215] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:29:43 (1776871783) [ 2313.184483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2313.187069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.190623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2316.256652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.256723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2316.264120] Lustre: Skipped 2 previous similar messages [ 2317.415732] Lustre: server umount lustre-MDT0000 complete [ 2318.622667] LustreError: 150528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871855 with bad export cookie 17304772425649319528 [ 2318.623956] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2318.626156] LustreError: 150528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2321.376504] LustreError: 150571:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.376546] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.380782] LustreError: 150571:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2321.384359] Lustre: Skipped 1 previous similar message [ 2321.384885] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2323.424442] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2323.424441] LustreError: 150542:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.766698] Lustre: server umount lustre-MDT0001 complete [ 2340.320112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2340.371430] Lustre: server umount lustre-OST0000 complete [ 2343.402459] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 2346.516743] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 2358.417112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.518368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2358.530748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2358.570516] Lustre: lustre-MDT0000: new disk, initializing [ 2358.596259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.605353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2360.008932] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2364.790351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2364.831327] Lustre: 156754:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2364.861733] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2364.864790] Lustre: Skipped 1 previous similar message [ 2364.904808] Lustre: lustre-MDT0001: new disk, initializing [ 2364.950140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2364.953790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2366.302678] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2369.465957] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2370.700507] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2373.007034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2373.123607] Lustre: lustre-OST0000: new disk, initializing [ 2373.125743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2373.154119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2373.156828] Lustre: Skipped 1 previous similar message [ 2375.321588] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2376.061687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2376.065921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2376.098753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2378.599696] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2380.020580] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2380.105916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2381.685233] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2381.770837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2386.405840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2386.406579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2386.408097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2386.411065] LustreError: Skipped 1 previous similar message [ 2386.419465] Lustre: Skipped 1 previous similar message [ 2388.776582] Lustre: server umount lustre-OST0000 complete [ 2391.523459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2391.531091] Lustre: Skipped 1 previous similar message [ 2396.642483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2396.647356] Lustre: Skipped 4 previous similar messages [ 2396.736249] Lustre: server umount lustre-MDT0000 complete [ 2398.372163] LustreError: 156748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871935 with bad export cookie 17304772425649478344 [ 2398.377794] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2398.379787] LustreError: 156748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2398.519190] Lustre: server umount lustre-MDT0001 complete [ 2402.162202] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:32:17 (1776871937) [ 2408.411166] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 2411.447798] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 2423.064192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.155848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2423.168229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2423.211823] Lustre: lustre-MDT0000: new disk, initializing [ 2423.234953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.241340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2424.560382] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2428.805262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.846254] Lustre: 163070:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2428.947216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2430.281435] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2433.338913] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2434.555983] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2437.156102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2438.584660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2438.618701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2439.161420] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2442.235765] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2448.867802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2448.867915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.871054] LustreError: Skipped 1 previous similar message [ 2448.876072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2448.878449] Lustre: Skipped 1 previous similar message [ 2450.189649] Lustre: server umount lustre-OST0000 complete [ 2457.638132] Lustre: server umount lustre-MDT0000 complete [ 2458.837337] LustreError: 163062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871995 with bad export cookie 17304772425649479212 [ 2458.840759] LustreError: 163062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2458.840905] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2458.944168] Lustre: server umount lustre-MDT0001 complete [ 2461.561953] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:33:17 (1776871997) [ 2467.215261] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 2470.107994] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 2481.658294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2481.750038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2481.761137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2481.763148] Lustre: Skipped 3 previous similar messages [ 2481.797147] Lustre: lustre-MDT0000: new disk, initializing [ 2481.798541] Lustre: Skipped 2 previous similar messages [ 2481.822402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.824459] Lustre: Skipped 2 previous similar messages [ 2481.829655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2481.832761] Lustre: Skipped 2 previous similar messages [ 2483.281491] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2487.897536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.954688] Lustre: 168988:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2488.061248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2489.490838] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2492.766947] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2494.026707] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2496.335672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2498.269222] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2501.375205] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2502.182775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2512.355325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2512.355523] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2512.358972] Lustre: Skipped 3 previous similar messages [ 2512.361232] LustreError: Skipped 1 previous similar message [ 2512.361626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2512.365551] Lustre: Skipped 3 previous similar messages [ 2514.770392] Lustre: server umount lustre-OST0000 complete [ 2530.272144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2530.335510] Lustre: server umount lustre-MDT0000 complete [ 2531.525143] LustreError: 168979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872068 with bad export cookie 17304772425649480283 [ 2531.528431] LustreError: 168979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2531.528786] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2531.639888] Lustre: server umount lustre-MDT0001 complete [ 2535.117763] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 2536.106146] Key type lgssc unregistered [ 2536.222466] LNet: 172466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2536.225627] LNetError: 172466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2536.234319] LNet: Removed LNI 192.168.204.137@tcp [ 2536.513526] Key type .llcrypt unregistered [ 2536.514837] Key type ._llcrypt unregistered [ 2545.287077] Key type ._llcrypt registered [ 2545.288199] Key type .llcrypt registered [ 2545.324371] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 2545.676619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2545.718527] alg: No test for adler32 (adler32-zlib) [ 2546.569191] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 2546.656236] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 2548.240152] Key type lgssc registered [ 2548.593534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2551.136482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2551.142836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.316384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.656288] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2556.621275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2556.738160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2558.060407] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2559.854727] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2561.012758] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2563.171838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2563.291843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.271256] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2567.053653] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2576.616779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2592.226753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.230861] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2592.231168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.236106] Lustre: Skipped 1 previous similar message [ 2595.085944] Lustre: server umount lustre-OST0000 complete [ 2597.345613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2597.346545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2597.349601] Lustre: Skipped 2 previous similar messages [ 2597.355820] Lustre: Skipped 1 previous similar message [ 2602.461280] Lustre: server umount lustre-MDT0000 complete [ 2602.465523] LustreError: 175210:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2602.471389] LustreError: 175210:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2603.742617] LustreError: 173821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872140 with bad export cookie 1541068120936323836 [ 2603.746502] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2603.746845] LustreError: 173821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2603.872683] Lustre: server umount lustre-MDT0001 complete [ 2607.574815] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 2608.652878] Key type lgssc unregistered [ 2608.785419] LNet: 177040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2608.787776] LNetError: 177040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2608.795332] LNet: Removed LNI 192.168.204.137@tcp [ 2609.110450] Key type .llcrypt unregistered [ 2609.112155] Key type ._llcrypt unregistered [ 2618.035126] Key type ._llcrypt registered [ 2618.036197] Key type .llcrypt registered [ 2618.073112] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 2618.421810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2618.453325] alg: No test for adler32 (adler32-zlib) [ 2619.333662] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 2619.427292] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 2621.016232] Key type lgssc registered [ 2621.444857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2624.053956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2624.060750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.241459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2626.573981] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2629.432520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.564857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2630.987337] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2632.849437] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2634.149612] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2636.535933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2636.644872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2638.512229] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2640.293665] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2649.967292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2655.204298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2655.206204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2655.209143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2655.221258] Lustre: Skipped 1 previous similar message [ 2655.228861] Lustre: Skipped 1 previous similar message [ 2655.756153] Lustre: lustre-OST0000: Not available for connect from 192.168.204.37@tcp (stopping) [ 2659.557849] Lustre: server umount lustre-OST0000 complete [ 2662.340678] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff917d84786000.ost_server_uuid 50 [ 2663.220673] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2675.619304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2675.620302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2675.625315] Lustre: Skipped 1 previous similar message [ 2680.759226] Lustre: server umount lustre-MDT0000 complete [ 2680.802260] LustreError: 180523:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2680.810528] LustreError: 180523:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2682.482976] LustreError: 178406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872219 with bad export cookie 1900031457874530969 [ 2682.485616] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2682.488992] LustreError: 178406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2682.642742] Lustre: server umount lustre-MDT0001 complete [ 2686.684602] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:37:02 (1776872222) [ 2689.634910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.843923] LustreError: 181918:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2689.880481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2691.503317] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2695.138329] LustreError: 181919:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.172855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2695.335293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2697.134414] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2699.390464] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2700.824276] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2703.744762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2703.894913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2706.376084] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2708.496247] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2711.091345] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2711.170596] Lustre: lustre-OST0001: new disk, initializing [ 2711.173683] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2713.253190] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2715.303808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2715.650798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2715.654875] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2715.692069] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2716.507186] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2720.225161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2720.228809] LustreError: Skipped 1 previous similar message [ 2720.230656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2720.236567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2720.239715] Lustre: Skipped 1 previous similar message [ 2723.524662] Lustre: server umount lustre-OST0000 complete [ 2725.721207] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2725.807452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2725.858711] LustreError: 183511:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2725.865668] LustreError: 183511:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2727.434497] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2727.540348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2730.981552] LustreError: 183509:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2730.988554] LustreError: 183509:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2741.218265] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2741.218812] LustreError: 183511:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.221707] LustreError: Skipped 1 previous similar message [ 2741.221737] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.227583] LustreError: 183511:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 2741.229717] Lustre: Skipped 1 previous similar message [ 2741.239898] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2741.242927] Lustre: Skipped 3 previous similar messages [ 2744.018661] Lustre: server umount lustre-OST0001 complete [ 2746.338659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.342840] Lustre: Skipped 2 previous similar messages [ 2751.457246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2751.459666] Lustre: Skipped 3 previous similar messages [ 2751.518126] Lustre: server umount lustre-MDT0000 complete [ 2752.891170] LustreError: 181900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872289 with bad export cookie 1900031457874532026 [ 2752.893842] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2752.895885] LustreError: 181900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2753.024924] Lustre: server umount lustre-MDT0001 complete [ 2761.649413] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:38:17 (1776872297) [ 2763.978179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2764.073176] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2764.087549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2764.155738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2764.158601] Lustre: Skipped 1 previous similar message [ 2765.473838] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2768.388780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.415651] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2768.425895] Lustre: 187681:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2769.808739] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2771.769729] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2773.094482] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2775.419572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2775.476941] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2776.583582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 2777.359067] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2779.121936] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2781.227506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2781.250409] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2783.054465] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2784.948030] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2785.110447] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2786.785472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2786.785529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.789489] LustreError: Skipped 2 previous similar messages [ 2786.796978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2786.796990] Lustre: Skipped 1 previous similar message [ 2800.096165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2800.136265] Lustre: server umount lustre-OST0000 complete [ 2800.611652] LustreError: 190778:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2800.618800] LustreError: 190778:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2815.971296] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2815.975099] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2817.045019] Lustre: server umount lustre-OST0001 complete [ 2820.065367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2820.068314] Lustre: Skipped 9 previous similar messages [ 2824.620162] Lustre: server umount lustre-MDT0000 complete [ 2826.046426] LustreError: 187673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872362 with bad export cookie 1900031457874533412 [ 2826.049443] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2826.050984] LustreError: 187673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2826.179515] Lustre: server umount lustre-MDT0001 complete [ 2836.213699] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:39:31 (1776872371) [ 2842.696850] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 2846.471292] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 2860.955673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.079619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2861.103875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2861.154897] Lustre: lustre-MDT0000: new disk, initializing [ 2861.202039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2861.204977] Lustre: Skipped 3 previous similar messages [ 2861.214758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2862.774820] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2867.483684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.534763] Lustre: 195260:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2867.564458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2867.567279] Lustre: Skipped 1 previous similar message [ 2867.606681] Lustre: lustre-MDT0001: new disk, initializing [ 2867.657097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2867.661444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2869.145230] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2872.513455] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2873.976972] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2876.639645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2876.762900] Lustre: lustre-OST0000: new disk, initializing [ 2876.766462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2879.025099] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2882.045900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2882.050031] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2882.081144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2882.396755] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2883.964323] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2884.059893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2885.543575] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2885.634784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2887.141897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2887.142794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2887.147205] Lustre: Skipped 3 previous similar messages [ 2887.150355] LustreError: Skipped 3 previous similar messages [ 2887.152634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2887.156577] Lustre: Skipped 3 previous similar messages [ 2888.411115] Lustre: server umount lustre-OST0000 complete [ 2904.032165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2904.105215] Lustre: server umount lustre-MDT0000 complete [ 2905.673650] LustreError: 195252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872442 with bad export cookie 1900031457874535176 [ 2905.677519] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2905.679629] LustreError: 195252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2905.802918] Lustre: server umount lustre-MDT0001 complete [ 2908.987847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2909.144604] LustreError: 199067:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2909.153202] LustreError: 199067:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 8 previous similar messages [ 2910.597760] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2913.457385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2914.942934] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2916.813255] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2918.066924] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2920.340303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2922.296051] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2924.052557] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2925.422918] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2934.695939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2936.125067] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2936.210476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2938.850864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2938.853274] LustreError: Skipped 1 previous similar message [ 2943.002877] Lustre: server umount lustre-OST0000 complete [ 2945.800487] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2948.528133] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2975.188838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2975.374948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.380239] Lustre: Skipped 5 previous similar messages [ 2978.303072] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2978.471531] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2978.477995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2978.478612] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2978.490995] Lustre: Skipped 1 previous similar message [ 2981.069787] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2995.487736] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 13 sec [ 2999.266579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2999.269927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2999.277330] Lustre: Skipped 5 previous similar messages [ 3002.833756] Lustre: server umount lustre-OST0000 complete [ 3010.465451] Lustre: server umount lustre-MDT0000 complete [ 3012.088575] LustreError: 199047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872548 with bad export cookie 1900031457874536051 [ 3012.091864] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3012.093539] LustreError: 199047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3012.215401] Lustre: server umount lustre-MDT0001 complete [ 3015.917217] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:42:31 (1776872551) [ 3018.551406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.718142] LustreError: 204591:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3020.302292] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3024.161512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3026.082515] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3028.531352] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3030.413435] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3033.713275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3036.582757] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3039.108659] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3040.849913] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3045.062324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3046.843290] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3050.026172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 3052.762366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3052.865462] Lustre: lustre-OST0001: new disk, initializing [ 3052.868876] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3055.423766] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3058.694685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3058.700039] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3058.740093] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3059.078407] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3060.630626] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3060.729420] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3062.407396] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3062.507425] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3063.778957] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3063.781837] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3063.782646] LustreError: Skipped 2 previous similar messages [ 3063.785754] Lustre: Skipped 14 previous similar messages [ 3065.354338] Lustre: server umount lustre-OST0001 complete [ 3067.912939] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3068.012039] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3069.563285] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3069.670818] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3094.278781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3096.953836] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3099.401559] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3099.622558] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3099.627844] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3099.627907] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3100.110279] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3107.021437] Lustre: server umount lustre-OST0001 complete [ 3125.588640] Lustre: server umount lustre-OST0000 complete [ 3130.338622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.343771] Lustre: Skipped 10 previous similar messages [ 3133.145952] Lustre: server umount lustre-MDT0000 complete [ 3134.481052] LustreError: 207633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872671 with bad export cookie 1900031457874537003 [ 3134.481536] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3134.486024] LustreError: 207633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3140.679463] Lustre: server umount lustre-MDT0001 complete [ 3151.450442] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:44:47 (1776872687) [ 3154.428485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3154.565705] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3154.582605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3154.674188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3154.677107] Lustre: Skipped 5 previous similar messages [ 3156.428113] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3160.284474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.334716] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3160.349092] Lustre: 212575:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3162.042280] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3164.248753] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3165.797644] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3168.550364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3168.626058] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3170.843302] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3173.048322] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3177.426044] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3177.486500] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3180.110758] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3182.576069] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3183.293619] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3185.028714] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3185.132538] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3186.856317] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3186.945773] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3188.468614] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff917d83edc800.ost_server_uuid 50 [ 3189.221370] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff917d83edc800.ost_server_uuid in FULL state after 0 sec [ 3189.631141] Lustre: Permanently deactivating lustre-OST0001 [ 3189.639696] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3190.684672] Lustre: Permanently reactivating lustre-OST0001 [ 3193.320018] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3193.324147] LustreError: Skipped 3 previous similar messages [ 3198.093968] Lustre: server umount lustre-OST0001 complete [ 3198.435363] LustreError: 214191:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3198.443058] LustreError: 214191:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 25 previous similar messages [ 3205.781372] Lustre: server umount lustre-OST0000 complete [ 3213.363979] Lustre: server umount lustre-MDT0000 complete [ 3215.163529] LustreError: 215106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872751 with bad export cookie 1900031457874538277 [ 3215.166255] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3215.174625] LustreError: 215106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3215.365313] Lustre: server umount lustre-MDT0001 complete [ 3221.865026] hrtimer: interrupt took 2981851 ns [ 3231.687492] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:46:07 (1776872767) [ 3237.382251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3237.635874] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3237.652300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3237.657529] Lustre: Skipped 5 previous similar messages [ 3241.064970] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3247.041100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3247.107100] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3247.128543] Lustre: 218676:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3249.538151] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3252.591216] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3254.668837] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3258.388631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3258.551294] Lustre: Permanently deactivating lustre-OST0000 [ 3258.561730] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3261.818856] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3262.164383] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3264.857183] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3268.516283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3268.988969] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3271.995556] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3274.668477] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3275.523937] Lustre: Permanently reactivating lustre-OST0000 [ 3275.534555] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 3275.538929] Lustre: Skipped 2 previous similar messages [ 3277.953974] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3277.959347] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3277.965822] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3277.968934] Lustre: Skipped 1 previous similar message [ 3277.987739] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3277.999952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3291.487394] Lustre: server umount lustre-OST0001 complete [ 3299.424662] Lustre: server umount lustre-OST0000 complete [ 3307.700495] Lustre: server umount lustre-MDT0000 complete [ 3309.488928] LustreError: 218669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872846 with bad export cookie 1900031457874539915 [ 3309.489444] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3309.495303] LustreError: 218669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3309.690243] Lustre: server umount lustre-MDT0001 complete [ 3314.747883] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 3316.046581] Key type lgssc unregistered [ 3316.227578] LNet: 223058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3316.232912] LNetError: 223058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3316.248188] LNet: Removed LNI 192.168.204.137@tcp [ 3316.619598] Key type .llcrypt unregistered [ 3316.621454] Key type ._llcrypt unregistered [ 3327.933891] Key type ._llcrypt registered [ 3327.935110] Key type .llcrypt registered [ 3327.991884] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 3328.400785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3328.407798] alg: No test for adler32 (adler32-zlib) [ 3329.308827] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 3329.417490] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 3331.016238] Key type lgssc registered [ 3331.544716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3336.237761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3336.247677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.486244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3339.039034] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3342.788145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.855593] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3342.872480] Lustre: 224474:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3342.879991] Lustre: Permanently deactivating lustre-MDT0001 [ 3342.888291] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3343.027592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3344.050611] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3344.055833] LustreError: 224516:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3344.064473] LustreError: 224516:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3344.070280] LustreError: 224516:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 3344.816705] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3346.985311] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3348.519459] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3351.603386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3351.772057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3352.806337] LustreError: 226098:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3352.813317] LustreError: 226098:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3352.815965] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3354.603654] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3357.115625] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3358.184899] LustreError: 226099:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3360.322279] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3360.448489] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3362.487951] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3362.488943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3362.500036] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3363.403794] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3366.244668] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3368.228078] Lustre: Permanently reactivating lustre-MDT0001 [ 3368.235685] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3368.239362] Lustre: Skipped 2 previous similar messages [ 3373.073725] Lustre: lustre-MDT0001: Client 79d1a8f5-5fe5-4395-8b38-4924a69d92bd (at 192.168.204.37@tcp) reconnecting [ 3375.276914] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3376.064619] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3378.490961] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3378.588679] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3380.387486] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3380.489869] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3381.790295] Lustre: Permanently deactivating lustre-MDT0001 [ 3381.800442] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3381.803880] Lustre: Skipped 2 previous similar messages [ 3387.922315] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3392.995632] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3393.000911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3393.008646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3393.507397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3393.509914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3393.514037] Lustre: Skipped 1 previous similar message [ 3393.520586] Lustre: Skipped 3 previous similar messages [ 3396.640403] Lustre: server umount lustre-MDT0000 complete [ 3398.300731] LustreError: 227576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872934 with bad export cookie 16117125050420843398 [ 3398.302128] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3398.306505] LustreError: 227576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3398.473720] Lustre: server umount lustre-MDT0001 complete [ 3406.336689] Lustre: server umount lustre-OST0000 complete [ 3411.155800] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:49:06 (1776872946) [ 3415.777418] Lustre: server umount lustre-OST0001 complete [ 3418.107272] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 3421.461454] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 3436.694244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3436.810272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3436.816578] Lustre: Skipped 2 previous similar messages [ 3436.833686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3436.882304] Lustre: lustre-MDT0000: new disk, initializing [ 3436.923695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3436.933258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3438.651627] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3443.798868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3443.853890] Lustre: 232112:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3443.882183] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3443.884711] Lustre: Skipped 1 previous similar message [ 3443.923964] Lustre: lustre-MDT0001: new disk, initializing [ 3443.978905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3443.984976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3445.611463] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3449.288328] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3450.738153] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3453.652493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3453.770172] Lustre: lustre-OST0000: new disk, initializing [ 3453.773144] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3453.808235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3453.811267] Lustre: Skipped 1 previous similar message [ 3456.201613] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3459.745640] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3463.183195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3463.187889] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3463.222173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3465.802825] LustreError: 232120:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3470.238040] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3470.307331] Lustre: lustre-OST0001: new disk, initializing [ 3470.309467] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3470.346063] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3472.400403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3472.405504] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3472.447205] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3472.573653] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3475.896077] LustreError: 232120:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3475.906815] LustreError: 232121:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3476.085880] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3476.808675] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3486.000140] LustreError: 232121:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3487.719202] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3487.721718] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3487.724192] LustreError: Skipped 1 previous similar message [ 3487.726027] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3487.733277] Lustre: Skipped 2 previous similar messages [ 3489.801082] Lustre: lustre-OST0001: Not available for connect from 192.168.204.37@tcp (stopping) [ 3489.803954] Lustre: Skipped 1 previous similar message [ 3492.932146] Lustre: server umount lustre-OST0001 complete [ 3497.955943] LustreError: 235513:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3497.962100] LustreError: 235513:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3503.078584] LustreError: 233839:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3503.085530] LustreError: 233839:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3508.195040] LustreError: 233839:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3508.195634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3508.197747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3508.197753] Lustre: Skipped 2 previous similar messages [ 3508.202368] LustreError: 233839:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3508.207971] Lustre: Skipped 1 previous similar message [ 3511.407065] Lustre: server umount lustre-OST0000 complete [ 3513.313972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.319654] Lustre: Skipped 1 previous similar message [ 3518.433753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3518.437150] Lustre: Skipped 4 previous similar messages [ 3519.084655] Lustre: server umount lustre-MDT0000 complete [ 3520.666894] LustreError: 232891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873057 with bad export cookie 16117125050420846205 [ 3520.670099] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3520.671898] LustreError: 232891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3526.874639] Lustre: server umount lustre-MDT0001 complete [ 3531.573110] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 3532.834774] Key type lgssc unregistered [ 3532.992403] LNet: 236760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3532.996087] LNetError: 236760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3533.004537] LNet: Removed LNI 192.168.204.137@tcp [ 3533.356479] Key type .llcrypt unregistered [ 3533.358187] Key type ._llcrypt unregistered [ 3550.567454] Key type ._llcrypt registered [ 3550.570030] Key type .llcrypt registered [ 3550.624384] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 3551.077630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3551.102128] alg: No test for adler32 (adler32-zlib) [ 3551.984708] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 3552.092236] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 3553.689328] Key type lgssc registered [ 3554.147922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3557.322156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3557.330374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3558.461224] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3558.473169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3558.546091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3560.144077] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3563.419849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3563.460492] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3563.476352] Lustre: 238926:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3563.584649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3565.151967] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3567.406642] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3568.978654] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3571.742676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3571.847308] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3571.949534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3573.786668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3574.291976] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3576.566074] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3619.809722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3619.810759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3619.815557] LustreError: Skipped 1 previous similar message [ 3619.822835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3619.827087] Lustre: Skipped 1 previous similar message [ 3622.195987] Lustre: server umount lustre-OST0000 complete [ 3623.983744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3628.935514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3629.086762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3631.392551] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3633.646717] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3640.360249] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3640.366937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3640.367223] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3640.375483] Lustre: Skipped 1 previous similar message [ 3671.011521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3671.017058] Lustre: Skipped 1 previous similar message [ 3671.019434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3671.020094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3671.026430] Lustre: Skipped 1 previous similar message [ 3673.442857] Lustre: server umount lustre-OST0000 complete [ 3676.130636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3676.131334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3676.131936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3676.133912] LustreError: Skipped 1 previous similar message [ 3676.139221] Lustre: Skipped 2 previous similar messages [ 3681.057340] Lustre: server umount lustre-MDT0000 complete [ 3682.459728] LustreError: 238920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873219 with bad export cookie 9533330100648024888 [ 3682.463900] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3682.465078] LustreError: 238920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3682.601686] Lustre: server umount lustre-MDT0001 complete [ 3686.740036] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 3687.940734] Key type lgssc unregistered [ 3688.092067] LNet: 243872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3688.096302] LNetError: 243872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3688.104965] LNet: Removed LNI 192.168.204.137@tcp [ 3688.462852] Key type .llcrypt unregistered [ 3688.464872] Key type ._llcrypt unregistered [ 3698.919513] Key type ._llcrypt registered [ 3698.920736] Key type .llcrypt registered [ 3698.960876] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 3699.320615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3699.355986] alg: No test for adler32 (adler32-zlib) [ 3700.238612] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 3700.342273] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 3701.928158] Key type lgssc registered [ 3702.406106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3705.559590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3705.569580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3706.769496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3708.276737] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3711.561436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.723995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3713.269085] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3715.380563] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3716.876435] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3719.576620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3719.719420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3721.917658] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3724.025648] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3732.004971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3747.300875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3747.301318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3747.305674] LustreError: Skipped 1 previous similar message [ 3747.312329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3747.315577] Lustre: Skipped 1 previous similar message [ 3749.595787] Lustre: server umount lustre-OST0000 complete [ 3752.418619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3752.419289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3752.422790] Lustre: Skipped 2 previous similar messages [ 3757.149333] Lustre: server umount lustre-MDT0000 complete [ 3757.538421] LustreError: 245249:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3757.546595] LustreError: 245249:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 3758.571648] LustreError: 245228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873295 with bad export cookie 12589732320929329024 [ 3758.574420] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3758.577076] LustreError: 245228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3758.682480] Lustre: server umount lustre-MDT0001 complete [ 3762.788274] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 3764.035752] Key type lgssc unregistered [ 3764.194202] LNet: 249210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3764.198179] LNetError: 249210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3764.206419] LNet: Removed LNI 192.168.204.137@tcp [ 3764.566880] Key type .llcrypt unregistered [ 3764.568597] Key type ._llcrypt unregistered [ 3772.329573] Key type ._llcrypt registered [ 3772.331522] Key type .llcrypt registered [ 3772.376535] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 3772.855595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3772.866979] alg: No test for adler32 (adler32-zlib) [ 3773.729753] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 3773.830931] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 3775.424141] Key type lgssc registered [ 3775.840717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3778.738800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3778.746561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3779.925243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3781.419994] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3784.599622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3784.744928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3786.194172] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3788.197551] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3789.555113] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3791.972538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3792.097946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3794.073467] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3796.094230] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3805.414206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3810.787865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3810.788049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3810.791228] LustreError: Skipped 1 previous similar message [ 3810.796192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3810.799423] Lustre: Skipped 1 previous similar message [ 3815.907179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3815.910405] Lustre: Skipped 1 previous similar message [ 3816.788815] Lustre: server umount lustre-OST0000 complete [ 3820.512795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3820.516439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.522060] Lustre: Skipped 1 previous similar message [ 3820.523837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3824.280140] Lustre: server umount lustre-MDT0000 complete [ 3825.638250] LustreError: 250374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873362 with bad export cookie 7109763635663647470 [ 3825.641625] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3825.643386] LustreError: 250374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3825.768058] Lustre: server umount lustre-MDT0001 complete [ 3829.643443] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 3830.765934] Key type lgssc unregistered [ 3830.903366] LNet: 253729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3830.907222] LNetError: 253729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3830.914377] LNet: Removed LNI 192.168.204.137@tcp [ 3831.252370] Key type .llcrypt unregistered [ 3831.254066] Key type ._llcrypt unregistered [ 3841.104198] Key type ._llcrypt registered [ 3841.105542] Key type .llcrypt registered [ 3841.151319] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 3841.685083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3841.699053] alg: No test for adler32 (adler32-zlib) [ 3842.571467] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 3842.668263] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 3844.256189] Key type lgssc registered [ 3844.646800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3847.461045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3847.468336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3848.635924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3850.032643] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3852.913637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3853.049989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3854.490461] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3856.379728] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3857.591731] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3859.865338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3859.982505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3862.026900] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3863.935001] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3875.367487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3888.098284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3888.101435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3888.105513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3888.608941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3888.611257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3888.616321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3890.657720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3890.659719] Lustre: Skipped 1 previous similar message [ 3893.846568] Lustre: server umount lustre-OST0000 complete [ 3895.779307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3895.784927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3895.787487] Lustre: Skipped 1 previous similar message [ 3900.897287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3900.900977] Lustre: Skipped 1 previous similar message [ 3901.273422] Lustre: server umount lustre-MDT0000 complete [ 3902.685384] LustreError: 256721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873439 with bad export cookie 14320845760463957715 [ 3902.689940] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3902.691036] LustreError: 256721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3902.803255] Lustre: server umount lustre-MDT0001 complete [ 3906.515024] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 3907.586363] Key type lgssc unregistered [ 3907.719417] LNet: 259074:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3907.722817] LNetError: 259074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3907.732501] LNet: Removed LNI 192.168.204.137@tcp [ 3908.051995] Key type .llcrypt unregistered [ 3908.053241] Key type ._llcrypt unregistered [ 3915.172246] Key type ._llcrypt registered [ 3915.173703] Key type .llcrypt registered [ 3915.212627] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 3915.547020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3915.580518] alg: No test for adler32 (adler32-zlib) [ 3916.443253] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 3916.530890] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 3918.120112] Key type lgssc registered [ 3918.457854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3920.915855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3920.921739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3922.078521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3923.290329] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3926.123347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3926.239348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3927.564712] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3929.269165] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3930.493102] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3932.779498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3932.903301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3934.762025] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3936.426489] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3947.238519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3956.705117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3956.707886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3956.713867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3957.730353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3957.731636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3957.731912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3957.742045] Lustre: Skipped 1 previous similar message [ 3961.812233] Lustre: server umount lustre-OST0000 complete [ 3967.456989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3967.459590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3967.464493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3969.312693] Lustre: server umount lustre-MDT0000 complete [ 3970.513155] LustreError: 260236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873507 with bad export cookie 16576215207891507495 [ 3970.516224] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3970.517468] LustreError: 260236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3970.635617] Lustre: server umount lustre-MDT0001 complete [ 3974.338328] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 3975.381938] Key type lgssc unregistered [ 3975.515487] LNet: 263633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3975.519050] LNetError: 263633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3975.526365] LNet: Removed LNI 192.168.204.137@tcp [ 3975.800489] Key type .llcrypt unregistered [ 3975.801665] Key type ._llcrypt unregistered [ 3979.362394] Key type ._llcrypt registered [ 3979.363791] Key type .llcrypt registered [ 3979.406599] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3984.835482] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 3990.995319] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 3991.471481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3991.486397] alg: No test for adler32 (adler32-zlib) [ 3992.347403] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 3992.438094] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 3994.016140] Key type lgssc registered [ 3994.352366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4005.466526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4005.472870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4006.564298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4006.576371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4006.612354] Lustre: lustre-MDT0000: new disk, initializing [ 4006.638129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4006.645804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4007.968347] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4012.220069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4012.263554] Lustre: 267105:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4012.289212] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4012.291505] Lustre: Skipped 1 previous similar message [ 4012.327412] Lustre: lustre-MDT0001: new disk, initializing [ 4012.359927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4012.373168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4012.377594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4013.733486] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4016.889221] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4018.108039] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4020.464982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4020.563487] Lustre: lustre-OST0000: new disk, initializing [ 4020.566124] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4020.609396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4022.471871] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4025.556709] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4025.848314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4025.852443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4025.867776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4026.974937] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4027.050722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4028.340575] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4028.416872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4030.946319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4030.946596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4030.948506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4030.949911] LustreError: Skipped 1 previous similar message [ 4030.954824] Lustre: Skipped 1 previous similar message [ 4035.152085] Lustre: server umount lustre-OST0000 complete [ 4038.112888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4038.115228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4038.119901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4038.122450] Lustre: Skipped 1 previous similar message [ 4041.185057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4041.185525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4042.525549] Lustre: server umount lustre-MDT0000 complete [ 4043.709121] LustreError: 267883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873580 with bad export cookie 9139928921773733834 [ 4043.712318] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4043.713382] LustreError: 267883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4043.820845] Lustre: server umount lustre-MDT0001 complete [ 4046.296358] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:59:42 (1776873582) [ 4048.352773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4048.499220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4049.684240] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4052.428656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4052.534481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4053.800771] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4055.478842] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4056.577260] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4058.769218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4060.566925] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4062.242403] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4083.680750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4083.683168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4083.687387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4083.689231] Lustre: Skipped 1 previous similar message [ 4086.988734] Lustre: server umount lustre-OST0000 complete [ 4088.288659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4088.291054] LustreError: Skipped 1 previous similar message [ 4088.292286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.295932] Lustre: Skipped 1 previous similar message [ 4088.297835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4088.299939] Lustre: Skipped 2 previous similar messages [ 4094.354359] Lustre: server umount lustre-MDT0000 complete [ 4094.945923] LustreError: 271125:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4094.953555] LustreError: 271125:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4095.542885] LustreError: 271089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873632 with bad export cookie 9139928921773734674 [ 4095.545713] LustreError: 271828:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4095.546242] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4095.546512] LustreError: 271089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4095.657174] Lustre: server umount lustre-MDT0001 complete [ 4098.879106] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 4099.819295] Key type lgssc unregistered [ 4099.938351] LNet: 274305:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4099.940914] LNetError: 274305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4099.948327] LNet: Removed LNI 192.168.204.137@tcp [ 4100.223866] Key type .llcrypt unregistered [ 4100.224886] Key type ._llcrypt unregistered [ 4108.602859] Key type ._llcrypt registered [ 4108.604376] Key type .llcrypt registered [ 4108.646536] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 4114.614789] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 4115.016976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4115.026077] alg: No test for adler32 (adler32-zlib) [ 4115.864284] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 4115.952516] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 4117.536198] Key type lgssc registered [ 4117.888168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4132.022879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4132.027637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4133.116550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4133.127208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4133.160543] Lustre: lustre-MDT0000: new disk, initializing [ 4133.184249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4133.190578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4134.399557] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4138.544358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4138.582134] Lustre: 278078:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4138.605337] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4138.607591] Lustre: Skipped 1 previous similar message [ 4138.639785] Lustre: lustre-MDT0001: new disk, initializing [ 4138.671275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4138.683515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4138.687817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4139.977924] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4143.070780] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4144.282606] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4146.495086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4146.589240] Lustre: lustre-OST2710: new disk, initializing [ 4146.591933] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4146.616818] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4147.444931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4147.449452] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4147.482919] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4148.482075] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4151.572755] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4153.864901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4153.921188] Lustre: lustre-OST03e8: new disk, initializing [ 4153.923622] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4153.953571] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4155.892678] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4157.312573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4157.316666] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4157.341075] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4157.667029] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4160.376088] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4160.448599] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4161.790111] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4161.863145] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4163.095849] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4163.163960] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4164.397086] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4164.481931] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4167.651351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4167.652758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4167.656849] Lustre: Skipped 2 previous similar messages [ 4167.658916] Lustre: Skipped 1 previous similar message [ 4171.610911] Lustre: server umount lustre-MDT0000 complete [ 4172.811406] LustreError: 278069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873709 with bad export cookie 12871202627031057047 [ 4172.816599] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4172.816760] LustreError: 278069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4172.917897] Lustre: server umount lustre-MDT0001 complete [ 4184.069331] Lustre: server umount lustre-OST2710 complete [ 4194.588951] Lustre: server umount lustre-OST03e8 complete [ 4200.633557] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 4203.850731] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 4216.698273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4216.788035] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4216.800430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4216.842525] Lustre: lustre-MDT0000: new disk, initializing [ 4216.870584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4216.879188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4218.198947] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4222.361971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4222.404351] Lustre: 285794:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4222.503992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4223.884059] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4226.978410] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4228.066687] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4230.319547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4230.424494] Lustre: lustre-OST0000: new disk, initializing [ 4230.426630] Lustre: Skipped 1 previous similar message [ 4230.428562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4230.430658] Lustre: Skipped 2 previous similar messages [ 4230.455525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4230.458744] Lustre: Skipped 1 previous similar message [ 4232.339136] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4232.566448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4232.569457] Lustre: Skipped 1 previous similar message [ 4232.571547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4232.597320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4235.341447] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4236.621685] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4236.683956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4237.930490] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4238.001957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4242.913517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4242.915225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4242.915855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4242.915859] Lustre: Skipped 2 previous similar messages [ 4242.923426] Lustre: Skipped 2 previous similar messages [ 4244.690316] Lustre: server umount lustre-OST0000 complete [ 4248.033636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4248.033861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4248.034468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4248.034473] Lustre: Skipped 1 previous similar message [ 4248.036913] LustreError: Skipped 1 previous similar message [ 4248.041063] Lustre: Skipped 1 previous similar message [ 4253.152894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4253.154940] Lustre: Skipped 2 previous similar messages [ 4258.273308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4258.275405] Lustre: Skipped 1 previous similar message [ 4260.320124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4260.380294] Lustre: server umount lustre-MDT0000 complete [ 4261.603401] LustreError: 286572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873798 with bad export cookie 12871202627031058713 [ 4261.606608] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4261.609462] LustreError: 286572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4261.727828] Lustre: server umount lustre-MDT0001 complete [ 4264.594855] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:03:20 (1776873800) [ 4265.082860] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4265.606249] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:03:21 (1776873801) [ 4266.483915] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 4274.879187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4274.985821] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4274.992596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4275.054037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4276.305578] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4278.987029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4279.014319] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4279.024959] Lustre: 291223:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4280.320093] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4281.941289] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4283.042098] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4298.147145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4298.208975] Lustre: Denying initial registration attempt from nid 192.168.204.137@tcp, specified as failover [ 4298.213621] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4298.218297] LustreError: 292833:0:(tgt_mount.c:2564:server_fill_super()) Unable to start targets: -99 [ 4298.220906] LustreError: 292833:0:(tgt_mount.c:2082:server_put_super()) no obd lustre-OST0000 [ 4298.223321] LustreError: 292833:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 4298.244996] Lustre: server umount lustre-OST0000 complete [ 4298.247057] LustreError: 292833:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4301.794468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4301.795127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4301.798598] Lustre: Skipped 1 previous similar message [ 4306.451620] Lustre: server umount lustre-MDT0000 complete [ 4306.913147] LustreError: 292317:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4306.919573] LustreError: 292317:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 4307.606835] LustreError: 293204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873844 with bad export cookie 12871202627031059588 [ 4307.609544] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4307.610482] LustreError: 293204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4307.702969] Lustre: server umount lustre-MDT0001 complete [ 4311.166886] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 4312.149043] Key type lgssc unregistered [ 4312.276329] LNet: 293937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4312.278615] LNetError: 293937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4312.288258] LNet: Removed LNI 192.168.204.137@tcp [ 4312.557715] Key type .llcrypt unregistered [ 4312.559361] Key type ._llcrypt unregistered [ 4316.012970] Key type ._llcrypt registered [ 4316.014592] Key type .llcrypt registered [ 4316.053548] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 4316.412651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4316.442209] alg: No test for adler32 (adler32-zlib) [ 4317.282679] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 4317.360523] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 4318.936118] Key type lgssc registered [ 4319.289343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4353.473632] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 4356.117335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4356.124432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4357.214273] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4357.221941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4357.284363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4358.584085] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4361.502154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4361.535103] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4361.545384] Lustre: 296646:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4361.656811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4362.970465] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4364.750987] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4366.007118] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4381.282488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4381.345178] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4381.400917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4383.117960] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4384.771026] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4385.486723] Lustre: server umount lustre-OST0000 complete [ 4386.785113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4386.785775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4386.789976] Lustre: Skipped 1 previous similar message [ 4386.793573] Lustre: Skipped 1 previous similar message [ 4391.405473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4391.407809] Lustre: Skipped 1 previous similar message [ 4392.793704] Lustre: server umount lustre-MDT0000 complete [ 4393.926124] LustreError: 297372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873930 with bad export cookie 1148354568063671307 [ 4393.929593] LustreError: 297372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4393.930509] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4394.042878] Lustre: server umount lustre-MDT0001 complete [ 4397.325211] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 4398.317700] Key type lgssc unregistered [ 4398.443406] LNet: 299852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4398.446273] LNetError: 299852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4398.453266] LNet: Removed LNI 192.168.204.137@tcp [ 4398.725115] Key type .llcrypt unregistered [ 4398.726232] Key type ._llcrypt unregistered [ 4407.344189] Key type ._llcrypt registered [ 4407.345324] Key type .llcrypt registered [ 4407.381625] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 4407.712985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4407.733784] alg: No test for adler32 (adler32-zlib) [ 4408.593661] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 4408.673658] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 4410.256103] Key type lgssc registered [ 4410.606267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4413.243936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4413.250026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4414.413767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4415.722561] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4418.618371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4418.746765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4420.070199] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4421.918875] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4423.209172] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4427.491517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4427.614388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4429.438352] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4431.121326] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4444.641115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4444.644426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4444.648593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4445.155391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4445.155498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4445.159596] Lustre: Skipped 1 previous similar message [ 4445.163384] Lustre: Skipped 2 previous similar messages [ 4446.726256] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.37@tcp (stopping) [ 4449.819262] Lustre: server umount lustre-MDT0000 complete [ 4450.273049] LustreError: 301248:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4450.279018] LustreError: 301248:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 4451.103215] LustreError: 301212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873987 with bad export cookie 12042171741927847813 [ 4451.107943] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4451.837685] LustreError: 301224:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4451.847570] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.37@tcp (stopping) [ 4455.393462] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4455.393749] LustreError: 301224:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4456.417345] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4456.419371] Lustre: Skipped 1 previous similar message [ 4457.271425] Lustre: server umount lustre-MDT0001 complete [ 4459.629180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4462.350900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4475.872471] LustreError: 300541:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005ce484b9 x1863187373777152/t0(0) o250->MGC192.168.204.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4475.973940] LustreError: 302832:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.980508] LustreError: 302832:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4475.996060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4476.020689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4477.221406] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4480.034841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4480.151331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4481.188796] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4481.393138] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4483.106084] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4486.307984] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4487.701887] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 4488.826222] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4489.390500] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4490.720944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4490.723055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4490.726532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4490.728951] Lustre: Skipped 1 previous similar message [ 4496.714907] Lustre: server umount lustre-OST0000 complete [ 4500.960874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4500.963628] LustreError: Skipped 1 previous similar message [ 4500.965297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4500.970342] Lustre: Skipped 1 previous similar message [ 4507.104931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4507.107038] Lustre: Skipped 6 previous similar messages [ 4512.224118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4512.290046] Lustre: server umount lustre-MDT0000 complete [ 4513.515218] LustreError: 301210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874050 with bad export cookie 12042171741927854939 [ 4513.518187] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4513.519624] LustreError: 301210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4513.630225] Lustre: server umount lustre-MDT0001 complete [ 4517.156415] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 4518.168061] Key type lgssc unregistered [ 4518.299542] LNet: 306930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4518.303035] LNetError: 306930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4518.310283] LNet: Removed LNI 192.168.204.137@tcp [ 4518.610431] Key type .llcrypt unregistered [ 4518.611665] Key type ._llcrypt unregistered [ 4527.567504] Key type ._llcrypt registered [ 4527.568466] Key type .llcrypt registered [ 4527.599990] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 4527.908324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4527.947252] alg: No test for adler32 (adler32-zlib) [ 4528.795391] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 4528.870089] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 4530.464180] Key type lgssc registered [ 4530.854413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4533.449933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4533.455766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4534.601955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4535.924131] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4538.805467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4538.938606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4540.347392] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4542.232661] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4543.483233] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4548.577747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4548.582043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4548.583794] Lustre: Skipped 1 previous similar message [ 4550.610710] Lustre: server umount lustre-MDT0000 complete [ 4551.765398] LustreError: 308288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874088 with bad export cookie 10386429989453235629 [ 4551.768261] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4551.770039] LustreError: 308288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4551.876180] Lustre: server umount lustre-MDT0001 complete [ 4556.484042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4556.575406] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4556.587163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4556.652695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4557.853676] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4560.436939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4560.462910] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4560.474635] Lustre: 310663:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4560.558580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4561.816338] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4563.479495] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4564.566836] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4568.931641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4569.001046] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4569.004992] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4569.009089] LustreError: 312319:0:(tgt_mount.c:2564:server_fill_super()) Unable to start targets: -2 [ 4569.011821] LustreError: 312319:0:(tgt_mount.c:2082:server_put_super()) no obd lustre-OST0000 [ 4569.014010] LustreError: 312319:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 4569.038293] Lustre: server umount lustre-OST0000 complete [ 4569.039866] LustreError: 312319:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4573.923536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4573.992429] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4574.057133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4576.008079] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4577.893525] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4580.582501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4580.614961] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4580.640548] Lustre: lustre-OST0001: new disk, initializing [ 4580.643042] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4582.602694] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4582.988200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4582.991651] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4583.031596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4583.065191] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4584.516524] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4588.515076] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4588.517589] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4588.518965] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4588.521092] Lustre: Skipped 2 previous similar messages [ 4591.619321] Lustre: server umount lustre-OST0001 complete [ 4593.121052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4593.123670] LustreError: Skipped 1 previous similar message [ 4593.125084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4593.129499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4593.132539] Lustre: Skipped 1 previous similar message [ 4593.635057] LustreError: 312787:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4593.640463] LustreError: 312787:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4598.754519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4598.754735] LustreError: 314430:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4598.756586] Lustre: Skipped 2 previous similar messages [ 4598.761195] LustreError: 314430:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4599.055385] Lustre: server umount lustre-OST0000 complete [ 4601.824927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4601.827506] LustreError: Skipped 1 previous similar message [ 4601.828797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4601.832962] Lustre: Skipped 1 previous similar message [ 4603.873228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4603.875116] Lustre: Skipped 2 previous similar messages [ 4606.430594] Lustre: server umount lustre-MDT0000 complete [ 4607.750425] LustreError: 310654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874144 with bad export cookie 10386429989453236168 [ 4607.752404] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4607.755619] LustreError: 310654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4607.871047] Lustre: server umount lustre-MDT0001 complete [ 4611.188613] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 4612.107372] Key type lgssc unregistered [ 4612.232315] LNet: 315567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4612.234593] LNetError: 315567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4612.248286] LNet: Removed LNI 192.168.204.137@tcp [ 4612.531736] Key type .llcrypt unregistered [ 4612.532763] Key type ._llcrypt unregistered [ 4642.601549] Key type ._llcrypt registered [ 4642.602527] Key type .llcrypt registered [ 4642.637099] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 4648.353621] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 4648.799265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4648.813890] alg: No test for adler32 (adler32-zlib) [ 4649.665980] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 4649.739867] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 4651.312076] Key type lgssc registered [ 4651.635488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4662.471555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4662.476675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4663.561089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4663.572306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4663.607505] Lustre: lustre-MDT0000: new disk, initializing [ 4663.627732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4663.633748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4664.849603] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4668.952691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4668.995556] Lustre: 320436:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4669.018578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4669.020716] Lustre: Skipped 1 previous similar message [ 4669.056090] Lustre: lustre-MDT0001: new disk, initializing [ 4669.087171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4669.099974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4669.104809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4670.394249] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4673.373200] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4674.528759] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4676.617827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4676.719280] Lustre: lustre-OST0000: new disk, initializing [ 4676.721462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4676.745639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4678.538823] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4681.509675] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4682.742566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4682.745716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4682.779569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4682.841191] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4682.916210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4684.210852] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4684.287102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4687.841510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4687.842274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4687.844205] LustreError: Skipped 1 previous similar message [ 4687.848441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4687.850916] Lustre: Skipped 1 previous similar message [ 4691.023234] Lustre: server umount lustre-OST0000 complete [ 4692.961143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4692.961600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4692.964779] Lustre: Skipped 2 previous similar messages [ 4698.080983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4698.082805] Lustre: Skipped 2 previous similar messages [ 4698.332325] Lustre: server umount lustre-MDT0000 complete [ 4699.477111] LustreError: 320428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874236 with bad export cookie 9171617217767428266 [ 4699.479149] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4699.481406] LustreError: 320428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4699.597485] Lustre: server umount lustre-MDT0001 complete [ 4702.083160] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:10:38 (1776874238) [ 4704.296095] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:10:40 (1776874240) [ 4706.821653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4706.968738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4708.119117] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4710.656041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4712.044764] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4713.769670] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4714.958627] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4717.241736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4717.364860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4717.368217] Lustre: Skipped 1 previous similar message [ 4719.196590] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4720.971620] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4739.913116] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4740.483967] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4741.013145] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4741.551601] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4742.195393] Lustre: Failing over lustre-MDT0000 [ 4742.345223] Lustre: server umount lustre-MDT0000 complete [ 4746.720773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4746.723437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4746.728271] LustreError: 324801:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4746.732594] LustreError: 324801:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4747.233399] LustreError: 326338:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4747.238674] LustreError: 326338:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4751.870307] LustreError: 326932:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4755.292182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4755.332313] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4755.418109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4755.438454] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4756.691113] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4756.989350] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4760.549695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4760.559756] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4760.578322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4761.791837] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4762.354884] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4764.207838] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4764.876418] Lustre: Failing over lustre-MDT0000 [ 4765.111509] Lustre: server umount lustre-MDT0000 complete [ 4765.665783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4765.666164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4765.667465] LustreError: 326339:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4765.667476] LustreError: 326339:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4765.682987] Lustre: Skipped 4 previous similar messages [ 4770.785023] LustreError: 324782:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.791298] LustreError: 324782:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 4777.927804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4777.965731] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4777.968658] LustreError: 328730:0:(ldlm_resource.c:1172:ldlm_resource_complain()) MGC192.168.204.137@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000f69cafce) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 4777.972762] LustreError: 324774:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 4778.053812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4778.076997] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4779.256898] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4782.089462] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4783.078502] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4783.080892] Lustre: Skipped 2 previous similar messages [ 4783.088690] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4783.103255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4784.147668] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4784.621968] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4788.195210] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4788.195861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4788.196705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4788.199219] Lustre: Skipped 2 previous similar messages [ 4792.910255] Lustre: server umount lustre-MDT0000 complete [ 4793.313373] LustreError: 324801:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4793.317946] LustreError: 324801:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 4793.999119] LustreError: 324762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874330 with bad export cookie 9171617217767430786 [ 4794.002030] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4794.002965] LustreError: 324762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4794.101983] Lustre: server umount lustre-MDT0001 complete [ 4804.440939] Lustre: server umount lustre-OST0000 complete [ 4809.413714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4809.557619] LustreError: 330565:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4810.884434] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4813.707935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4813.822141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4813.824436] Lustre: Skipped 1 previous similar message [ 4815.197599] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4816.957768] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4818.220515] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4820.316178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4822.200425] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4823.810396] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4834.789554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4844.000902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4844.004118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4844.009129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4844.011986] Lustre: Skipped 2 previous similar messages [ 4846.672163] Lustre: server umount lustre-OST0000 complete [ 4854.034634] Lustre: server umount lustre-MDT0000 complete [ 4855.264876] LustreError: 330566:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4855.271098] LustreError: 330566:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4855.299347] LustreError: 330545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874391 with bad export cookie 9171617217767431528 [ 4855.303141] LustreError: 330545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4855.303849] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4855.424549] Lustre: server umount lustre-MDT0001 complete [ 4859.052451] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 4860.073054] Key type lgssc unregistered [ 4860.208332] LNet: 333759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4860.210871] LNetError: 333759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4860.222350] LNet: Removed LNI 192.168.204.137@tcp [ 4860.507770] Key type .llcrypt unregistered [ 4860.509028] Key type ._llcrypt unregistered [ 4870.225108] Key type ._llcrypt registered [ 4870.226379] Key type .llcrypt registered [ 4870.273013] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 4870.766143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4870.781595] alg: No test for adler32 (adler32-zlib) [ 4871.641450] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 4871.739867] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 4873.328165] Key type lgssc registered [ 4873.716347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4876.343938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4876.349860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4877.504467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4878.817644] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4881.680096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4881.822847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4883.127934] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4884.863929] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4886.040214] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4888.232255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4888.356539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4890.231101] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4892.014213] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4902.693916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 4912.608830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4912.611462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4912.615843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4913.121387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4913.122748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4913.124355] Lustre: Skipped 1 previous similar message [ 4913.127929] Lustre: Skipped 1 previous similar message [ 4917.467181] Lustre: server umount lustre-MDT0000 complete [ 4918.241237] LustreError: 335236:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4918.246515] LustreError: 335236:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 4918.718210] LustreError: 336850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874455 with bad export cookie 15390021197923853631 [ 4918.721219] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4918.723465] LustreError: 336850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4918.845910] Lustre: server umount lustre-MDT0001 complete [ 4930.336117] Lustre: server umount lustre-OST0000 complete [ 4934.401648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4934.549467] LustreError: 338626:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4934.570572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4935.852316] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4938.638607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4938.762098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4940.063768] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4941.776564] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4942.946198] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4945.224827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4947.301771] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4949.054840] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4959.718181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4969.440935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4969.443637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4969.447679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4972.702712] Lustre: server umount lustre-MDT0000 complete [ 4973.925323] LustreError: 338608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874510 with bad export cookie 15390021197923854723 [ 4973.928613] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4973.928962] LustreError: 338608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4974.048992] Lustre: server umount lustre-MDT0001 complete [ 4984.540690] Lustre: server umount lustre-OST0000 complete [ 4989.540322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4989.683718] LustreError: 342215:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4989.705538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4989.707770] Lustre: Skipped 1 previous similar message [ 4990.947355] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4993.735172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4995.249911] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4997.132209] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4998.330350] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5000.709948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5002.892866] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5004.680459] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5015.206860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5024.737220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5024.739318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5024.743507] Lustre: Skipped 2 previous similar messages [ 5024.745231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5024.746824] Lustre: Skipped 2 previous similar messages [ 5027.408739] Lustre: server umount lustre-OST0000 complete [ 5029.856742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5029.858974] LustreError: Skipped 1 previous similar message [ 5029.860289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5029.863717] Lustre: Skipped 1 previous similar message [ 5029.865275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5029.867440] Lustre: Skipped 2 previous similar messages [ 5034.706914] Lustre: server umount lustre-MDT0000 complete [ 5035.921051] LustreError: 342198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874572 with bad export cookie 15390021197923855738 [ 5035.924331] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5035.926704] LustreError: 342198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5036.035601] Lustre: server umount lustre-MDT0001 complete [ 5039.471366] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5040.470851] Key type lgssc unregistered [ 5040.594355] LNet: 345411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5040.596929] LNetError: 345411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5040.606317] LNet: Removed LNI 192.168.204.137@tcp [ 5040.889421] Key type .llcrypt unregistered [ 5040.890550] Key type ._llcrypt unregistered [ 5049.895184] Key type ._llcrypt registered [ 5049.896348] Key type .llcrypt registered [ 5049.934852] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 5050.289845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5050.308283] alg: No test for adler32 (adler32-zlib) [ 5051.152465] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 5051.243415] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 5052.824198] Key type lgssc registered [ 5053.213841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5055.807816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5055.815624] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5055.819808] LustreError: 346813:0:(osd_handler.c:8532:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5055.830164] LustreError: 346813:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5055.832750] LustreError: 346813:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5055.835062] LustreError: 346813:0:(tgt_mount.c:2532:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5055.837542] LustreError: 346813:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5058.240237] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5058.243060] LustreError: 347209:0:(osd_handler.c:8532:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5058.252088] LustreError: 347209:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5058.255506] LustreError: 347209:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5058.259261] LustreError: 347209:0:(tgt_mount.c:2532:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5058.263285] LustreError: 347209:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5064.266110] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5065.284880] Key type lgssc unregistered [ 5065.407388] LNet: 348193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5065.410232] LNetError: 348193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5065.417270] LNet: Removed LNI 192.168.204.137@tcp [ 5065.718379] Key type .llcrypt unregistered [ 5065.719585] Key type ._llcrypt unregistered [ 5071.380658] Key type ._llcrypt registered [ 5071.382072] Key type .llcrypt registered [ 5071.422223] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 5077.702611] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 5078.020542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5078.063059] alg: No test for adler32 (adler32-zlib) [ 5078.916059] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 5079.010135] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 5080.592196] Key type lgssc registered [ 5080.969103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5092.731641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5092.736867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5093.833462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5093.846788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5093.885471] Lustre: lustre-MDT0000: new disk, initializing [ 5093.912374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5093.919670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5095.307543] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5099.510577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5099.554581] Lustre: 351288:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5099.578680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5099.581712] Lustre: Skipped 1 previous similar message [ 5099.626271] Lustre: lustre-MDT0001: new disk, initializing [ 5099.659970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5099.671511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5099.676638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5100.984263] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5104.119242] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5105.429125] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5107.730593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5107.838602] Lustre: lustre-OST0000: new disk, initializing [ 5107.840563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5107.866201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5109.722396] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5110.263981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5110.267959] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5110.302108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5112.747415] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5114.054875] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5114.124214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5115.451035] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5115.540289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5120.485737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5120.487319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5120.488062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5120.489820] Lustre: Skipped 1 previous similar message [ 5120.492795] LustreError: Skipped 1 previous similar message [ 5122.318972] Lustre: server umount lustre-OST0000 complete [ 5125.602700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5125.603290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5125.613515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5125.616804] Lustre: Skipped 2 previous similar messages [ 5129.682489] Lustre: server umount lustre-MDT0000 complete [ 5130.884135] LustreError: 352065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874667 with bad export cookie 4594771022186302000 [ 5130.886927] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5130.887743] LustreError: 352065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5130.997126] Lustre: server umount lustre-MDT0001 complete [ 5133.827636] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5134.396871] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:17:50 (1776874670) [ 5136.588959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5136.733976] LustreError: 355338:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5136.742374] LustreError: 355338:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5136.754856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5138.093852] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5140.867505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5140.996470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5142.347032] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5144.055932] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5145.213728] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5147.374924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5149.486952] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5151.390320] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5153.649495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5153.705967] Lustre: lustre-OST0001: new disk, initializing [ 5153.708059] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5153.735160] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5153.737305] Lustre: Skipped 1 previous similar message [ 5154.066233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5154.069487] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5154.106318] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5155.617915] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5158.807445] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5161.703640] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5161.708484] Lustre: Skipped 1 previous similar message [ 5161.711293] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5161.714165] Lustre: Skipped 1 previous similar message [ 5165.062460] Lustre: lustre-OST0001: Not available for connect from 192.168.204.37@tcp (stopping) [ 5166.783148] Lustre: server umount lustre-OST0001 complete [ 5167.075677] LustreError: 357908:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5167.080684] LustreError: 357908:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5172.196637] LustreError: 358526:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5172.201387] LustreError: 358526:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5177.317666] LustreError: 358526:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5177.326477] LustreError: 358526:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5181.408712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5181.411541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5181.416670] Lustre: Skipped 1 previous similar message [ 5181.418345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5182.435731] LustreError: 357908:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5182.440567] LustreError: 357908:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5192.673227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5192.675419] LustreError: 358526:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5192.676077] Lustre: Skipped 5 previous similar messages [ 5192.683288] LustreError: 358526:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5193.184092] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5193.234409] Lustre: server umount lustre-OST0000 complete [ 5197.280875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5197.283520] LustreError: Skipped 1 previous similar message [ 5197.284903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.288927] Lustre: Skipped 1 previous similar message [ 5200.608884] Lustre: server umount lustre-MDT0000 complete [ 5201.845166] LustreError: 355320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874738 with bad export cookie 4594771022186302868 [ 5201.847922] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5201.848700] LustreError: 355320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5216.224155] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5216.317968] Lustre: server umount lustre-MDT0001 complete [ 5219.592132] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5220.492233] Key type lgssc unregistered [ 5220.608418] LNet: 359687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5220.610590] LNetError: 359687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5220.619461] LNet: Removed LNI 192.168.204.137@tcp [ 5220.920329] Key type .llcrypt unregistered [ 5220.921596] Key type ._llcrypt unregistered [ 5237.410105] Key type ._llcrypt registered [ 5237.411199] Key type .llcrypt registered [ 5237.447416] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 5237.822634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5237.847816] alg: No test for adler32 (adler32-zlib) [ 5238.710897] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 5238.791366] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 5240.368179] Key type lgssc registered [ 5240.738790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5243.200964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5243.206742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5244.299516] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5244.309789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5244.371025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5245.765289] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5248.708749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5248.745856] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5248.756457] Lustre: 362442:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5248.857400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5250.208567] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5252.015306] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5253.297219] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5254.113068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5254.115636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5254.120066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5258.209131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5258.209596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5260.115618] Lustre: server umount lustre-MDT0000 complete [ 5261.321836] LustreError: 362478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874797 with bad export cookie 5552587841306575289 [ 5261.325371] LustreError: 362478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5261.326600] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5261.442555] Lustre: server umount lustre-MDT0001 complete [ 5264.457629] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:20:00 (1776874800) [ 5266.642614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5266.816689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5268.060301] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5270.764444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5270.880685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5272.153255] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5273.933369] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5275.118422] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5277.318767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5277.387717] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5277.447410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5279.292810] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5280.969263] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5289.921509] Lustre: Permanently deactivating lustre-OST0000 [ 5289.926522] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5292.270639] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5293.123766] LustreError: 367274:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5293.127183] LustreError: 367274:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 5293.129718] LustreError: 367274:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5293.747861] LustreError: 367322:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 5293.750336] LustreError: 367322:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5300.432909] Lustre: server umount lustre-OST0000 complete [ 5301.309875] LustreError: 367572:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 5301.312857] LustreError: 367572:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5303.265522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5303.265940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5303.269049] Lustre: Skipped 1 previous similar message [ 5303.271724] Lustre: Skipped 2 previous similar messages [ 5308.384964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5308.386950] Lustre: Skipped 1 previous similar message [ 5311.481294] Lustre: server umount lustre-MDT0000 complete [ 5312.759352] LustreError: 364708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874849 with bad export cookie 5552587841306575849 [ 5312.762471] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5312.764169] LustreError: 364708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5323.291044] Lustre: server umount lustre-MDT0001 complete [ 5326.070370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5327.371539] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5328.273279] LustreError: 368711:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5330.286316] Lustre: 369050:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.137@tcp [ 5330.911463] Lustre: server umount MGS complete [ 5334.661189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5334.805271] LustreError: 369772:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5334.809328] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5334.814355] LustreError: 369772:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5334.816283] Lustre: Skipped 1 previous similar message [ 5334.832405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5336.100057] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5339.005681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5339.096423] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5340.378794] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5342.150824] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5343.317562] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5343.925788] Lustre: Permanently reactivating lustre-OST0000 [ 5343.932372] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5343.934928] Lustre: Skipped 2 previous similar messages [ 5370.336076] LustreError: 371373:0:(osp_dev.c:695:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5373.284026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5373.399620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5373.403029] Lustre: Skipped 1 previous similar message [ 5375.357796] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5377.091853] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5389.794115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5389.796220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5389.797227] LustreError: Skipped 1 previous similar message [ 5389.803171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5389.807474] Lustre: Skipped 1 previous similar message [ 5391.250262] Lustre: server umount lustre-OST0000 complete [ 5394.578454] Lustre: server umount lustre-MDT0000 complete [ 5394.914428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5394.914966] LustreError: 370485:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5394.918957] Lustre: Skipped 2 previous similar messages [ 5394.924538] LustreError: 370485:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5395.802072] LustreError: 369799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874932 with bad export cookie 5552587841306576892 [ 5395.804872] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5395.805252] LustreError: 369799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5395.917521] Lustre: server umount lustre-MDT0001 complete [ 5399.478954] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5400.433919] Key type lgssc unregistered [ 5400.541324] LNet: 374131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5400.544387] LNetError: 374131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5400.552290] LNet: Removed LNI 192.168.204.137@tcp [ 5400.811683] Key type .llcrypt unregistered [ 5400.812868] Key type ._llcrypt unregistered [ 5406.706239] Key type ._llcrypt registered [ 5406.707873] Key type .llcrypt registered [ 5406.747797] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 5413.033159] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 5413.331932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5413.339543] alg: No test for adler32 (adler32-zlib) [ 5414.188807] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 5414.270471] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 5415.856125] Key type lgssc registered [ 5416.197770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5427.214449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5427.220596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5428.310409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5428.322314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5428.356030] Lustre: lustre-MDT0000: new disk, initializing [ 5428.380140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5428.386617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5429.720851] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5433.784063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5433.821991] Lustre: 377228:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5433.842246] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5433.844095] Lustre: Skipped 1 previous similar message [ 5433.878788] Lustre: lustre-MDT0001: new disk, initializing [ 5433.911781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5433.924423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5433.929336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5435.195349] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5438.379340] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5439.698424] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5442.085391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5442.183332] Lustre: lustre-OST0000: new disk, initializing [ 5442.185353] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5442.212131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5444.119442] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5445.495745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5445.498911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5445.526659] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5447.176394] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5448.496769] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5448.569533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5449.825530] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5449.906658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5450.723942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5450.725451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.729807] LustreError: Skipped 1 previous similar message [ 5450.730766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5450.739374] Lustre: Skipped 1 previous similar message [ 5452.755175] Lustre: server umount lustre-OST0000 complete [ 5454.304890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5454.307671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5454.311535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5454.313456] Lustre: Skipped 1 previous similar message [ 5455.841887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5455.842103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5460.122141] Lustre: server umount lustre-MDT0000 complete [ 5461.323269] LustreError: 377218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874997 with bad export cookie 3491104328725238942 [ 5461.326266] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5461.327230] LustreError: 377218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5461.438694] Lustre: server umount lustre-MDT0001 complete [ 5464.342987] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5464.883703] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:23:20 (1776875000) [ 5467.103123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5467.248035] LustreError: 381274:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5467.253307] LustreError: 381274:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5467.270981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5468.521415] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5471.403709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5471.524709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5472.841278] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5474.580837] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5475.809360] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5478.110947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5480.190332] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5481.890547] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5482.640454] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5493.339230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5496.801882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5496.802215] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5496.804655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5496.804659] Lustre: Skipped 1 previous similar message [ 5496.806026] Lustre: Skipped 1 previous similar message [ 5500.940065] Lustre: server umount lustre-OST0000 complete [ 5502.432749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5502.435395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5502.439896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5502.442164] Lustre: Skipped 1 previous similar message [ 5508.307655] Lustre: server umount lustre-MDT0000 complete [ 5509.463926] LustreError: 381255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776875046 with bad export cookie 3491104328725239810 [ 5509.465756] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5509.467392] LustreError: 381255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5509.581459] Lustre: server umount lustre-MDT0001 complete [ 5512.923421] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5513.969070] Key type lgssc unregistered [ 5514.104303] LNet: 384660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5514.106630] LNetError: 384660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5514.115261] LNet: Removed LNI 192.168.204.137@tcp [ 5514.391178] Key type .llcrypt unregistered [ 5514.392505] Key type ._llcrypt unregistered [ 5521.318155] Key type ._llcrypt registered [ 5521.319315] Key type .llcrypt registered [ 5521.348519] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5521.433151] Key type .llcrypt unregistered [ 5521.434105] Key type ._llcrypt unregistered [ 5527.846736] Key type ._llcrypt registered [ 5527.847782] Key type .llcrypt registered [ 5527.879093] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 5528.254242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5528.263948] alg: No test for adler32 (adler32-zlib) [ 5529.107829] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 5529.178230] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 5530.752135] Key type lgssc registered [ 5531.062385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5533.328225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5533.334559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5534.477391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5535.650196] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5538.216579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5538.315367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5540.035760] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5541.648441] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5543.393597] LustreError: 386813:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5543.399265] LustreError: 386813:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 5543.734831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5543.848746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5544.870509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5545.032131] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5549.025133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5549.027171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5549.030527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5550.050114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5550.050549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5550.051478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5553.812132] Lustre: server umount lustre-OST0000 complete [ 5555.169381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5555.169965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5555.174435] Lustre: Skipped 1 previous similar message [ 5555.176247] Lustre: Skipped 1 previous similar message [ 5560.289118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5560.291051] Lustre: Skipped 2 previous similar messages [ 5561.176566] Lustre: server umount lustre-MDT0000 complete [ 5562.400263] LustreError: 386793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776875099 with bad export cookie 15676474716897171422 [ 5562.403848] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5562.404588] LustreError: 386793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5562.521703] Lustre: server umount lustre-MDT0001 complete [ 5565.739454] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5566.696787] Key type lgssc unregistered [ 5566.838403] LNet: 389792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5566.841563] LNetError: 389792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5566.849413] LNet: Removed LNI 192.168.204.137@tcp [ 5567.168557] Key type .llcrypt unregistered [ 5567.169613] Key type ._llcrypt unregistered [ 5575.862601] Key type ._llcrypt registered [ 5575.863832] Key type .llcrypt registered [ 5575.903155] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 5576.252496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5576.268839] alg: No test for adler32 (adler32-zlib) [ 5577.125717] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 5577.203487] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 5578.784278] Key type lgssc registered [ 5579.119115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5581.692115] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5581.698373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5597.152235] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5602.272222] LustreError: 391148:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.137@tcp: failed processing log, type 4: rc = -110 [ 5628.896187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5630.847761] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5632.665961] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5635.038957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5635.242771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5635.287475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5636.593589] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5639.508020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5639.653689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5640.973169] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5642.765186] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5643.995724] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5648.866986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5648.868535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5648.871808] Lustre: Skipped 1 previous similar message [ 5652.305369] Lustre: server umount lustre-OST0000 complete [ 5653.985442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5653.986014] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5653.991294] Lustre: Skipped 1 previous similar message [ 5653.994464] Lustre: Skipped 1 previous similar message [ 5659.105098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5659.108015] Lustre: Skipped 1 previous similar message [ 5659.677892] Lustre: server umount lustre-MDT0000 complete [ 5660.906150] LustreError: 392778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776875197 with bad export cookie 12603710415552024357 [ 5660.909236] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5660.909890] LustreError: 392778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5661.020022] Lustre: server umount lustre-MDT0001 complete [ 5664.404899] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5665.368604] Key type lgssc unregistered [ 5665.496338] LNet: 394359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5665.498961] LNetError: 394359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5665.507304] LNet: Removed LNI 192.168.204.137@tcp [ 5665.787956] Key type .llcrypt unregistered [ 5665.789689] Key type ._llcrypt unregistered [ 5675.063713] Key type ._llcrypt registered [ 5675.065097] Key type .llcrypt registered [ 5675.103235] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 5675.439614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5675.452780] alg: No test for adler32 (adler32-zlib) [ 5676.323183] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 5676.406319] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 5677.992139] Key type lgssc registered [ 5678.381989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5681.009560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5681.018389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5682.163824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5683.369796] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5685.993638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5686.106064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5687.263184] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5688.953516] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5690.172878] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5692.425218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5692.537789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5694.293193] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5696.010324] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5707.878391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5713.380726] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.381483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5713.384261] Lustre: Skipped 2 previous similar messages [ 5713.385966] Lustre: Skipped 1 previous similar message [ 5716.995795] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.37@tcp (stopping) [ 5716.997874] Lustre: Skipped 1 previous similar message [ 5718.497938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5718.499756] Lustre: Skipped 1 previous similar message [ 5718.876092] Lustre: server umount lustre-MDT0000 complete [ 5719.617615] LustreError: 395722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.37@tcp arrived at 1776875256 with bad export cookie 18313669926292202290 [ 5719.621830] LustreError: 395722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5720.211893] LustreError: 395723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776875256 with bad export cookie 18313669926292201660 [ 5720.215439] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5720.215735] LustreError: 395723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5721.568947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5721.571177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5721.575089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5723.616915] LustreError: 395743:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5723.624689] LustreError: 395743:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5725.664816] LustreError: 395742:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5725.665090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5725.669357] LustreError: 395742:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5725.671055] Lustre: Skipped 1 previous similar message [ 5726.350384] Lustre: server umount lustre-OST0000 complete [ 5728.776755] Lustre: server umount lustre-MDT0001 complete [ 5732.628693] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5733.675178] Key type lgssc unregistered [ 5733.802558] LNet: 399091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5733.806761] LNetError: 399091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5733.819323] LNet: Removed LNI 192.168.204.137@tcp [ 5734.099647] Key type .llcrypt unregistered [ 5734.100748] Key type ._llcrypt unregistered [ 5742.688437] Key type ._llcrypt registered [ 5742.690059] Key type .llcrypt registered [ 5742.731326] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 5743.088629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5743.096704] alg: No test for adler32 (adler32-zlib) [ 5743.949482] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 5744.037465] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 5745.624189] Key type lgssc registered [ 5746.046730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5749.037190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5749.042906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5750.196959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5751.382988] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5754.485033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5754.602149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5755.931543] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5757.710973] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5758.844657] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5760.971799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5761.081278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5762.853992] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5764.672521] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5774.375192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5779.427181] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5779.431247] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5779.433192] Lustre: Skipped 1 previous similar message [ 5780.487163] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.37@tcp (stopping) [ 5782.351642] Lustre: server umount lustre-MDT0001 complete [ 5784.545492] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5784.545509] LustreError: 400468:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5784.546386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5784.546391] Lustre: Skipped 1 previous similar message [ 5784.546803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5784.558681] LustreError: 400468:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5789.664941] LustreError: 400467:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5789.665738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5790.028434] Lustre: server umount lustre-OST0000 complete [ 5791.331356] Lustre: server umount lustre-MDT0000 complete [ 5795.765416] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5796.679615] Key type lgssc unregistered [ 5796.809339] LNet: 403812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5796.811872] LNetError: 403812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5796.820285] LNet: Removed LNI 192.168.204.137@tcp [ 5797.099850] Key type .llcrypt unregistered [ 5797.100967] Key type ._llcrypt unregistered [ 5804.864397] Key type ._llcrypt registered [ 5804.865520] Key type .llcrypt registered [ 5804.900939] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 5810.819292] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 5811.126395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5811.154054] alg: No test for adler32 (adler32-zlib) [ 5811.997066] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 5812.074063] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 5813.648122] Key type lgssc registered [ 5814.027157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5825.990885] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 5831.196624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5831.202712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5832.288377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5832.301902] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5832.337611] Lustre: lustre-MDT0000: new disk, initializing [ 5832.360832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5832.366416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5833.573932] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5839.140953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5839.240921] Lustre: lustre-OST0000: new disk, initializing [ 5839.244101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5839.247236] Lustre: Skipped 1 previous similar message [ 5839.271069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5841.105747] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5841.343241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5841.346049] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5841.354030] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5844.056604] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5849.285113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5849.328148] Lustre: 407744:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5849.362252] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5849.397266] Lustre: lustre-MDT0001: new disk, initializing [ 5849.442865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5849.459200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 5849.462932] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 5850.742812] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5854.244482] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5854.314958] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5855.646751] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5855.720551] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5858.822873] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.37@tcp (stopping) [ 5859.348735] Lustre: server umount lustre-MDT0001 complete [ 5860.323053] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5860.323064] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5860.330016] Lustre: Skipped 1 previous similar message [ 5865.442239] LustreError: 407753:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5865.447129] LustreError: 407753:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 5870.562093] LustreError: 407782:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5870.566742] LustreError: 407782:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5872.096760] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5872.099073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5872.103345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5875.681501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5875.681976] LustreError: 407758:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5877.707458] Lustre: server umount lustre-OST0000 complete [ 5885.184157] Lustre: server umount lustre-MDT0000 complete [ 5889.305640] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5890.198752] Key type lgssc unregistered [ 5890.322305] LNet: 412309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5890.324548] LNetError: 412309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5890.333270] LNet: Removed LNI 192.168.204.137@tcp [ 5890.603472] Key type .llcrypt unregistered [ 5890.604494] Key type ._llcrypt unregistered [ 5902.617273] Key type ._llcrypt registered [ 5902.618361] Key type .llcrypt registered [ 5902.655691] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5902.754211] Key type .llcrypt unregistered [ 5902.755453] Key type ._llcrypt unregistered [ 5909.023732] Key type ._llcrypt registered [ 5909.024795] Key type .llcrypt registered [ 5909.062090] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 5909.510927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5909.527902] alg: No test for adler32 (adler32-zlib) [ 5910.388394] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 5910.478903] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 5912.064151] Key type lgssc registered [ 5912.485464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5921.746914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5921.752596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5922.838563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5922.848775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5922.888803] Lustre: lustre-MDT0000: new disk, initializing [ 5922.912614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5922.919533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5924.094503] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5928.126526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5928.163494] Lustre: 415627:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5928.183965] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5928.185948] Lustre: Skipped 1 previous similar message [ 5928.221225] Lustre: lustre-MDT0001: new disk, initializing [ 5928.250823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5928.261882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5928.265717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5929.535861] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5932.583893] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5933.774956] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5935.850235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5935.944257] Lustre: lustre-OST0000: new disk, initializing [ 5935.946157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5935.970406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5937.724260] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5940.677142] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5941.238877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5941.242254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5941.270807] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5948.896921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5948.899315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5948.903265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5951.461039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5951.461322] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5951.463261] Lustre: Skipped 1 previous similar message [ 5951.467455] Lustre: Skipped 1 previous similar message [ 5954.136564] Lustre: server umount lustre-MDT0000 complete [ 5955.337077] LustreError: 416403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776875491 with bad export cookie 8042375554046004133 [ 5955.340122] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5955.340377] LustreError: 416403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5955.451669] Lustre: server umount lustre-MDT0001 complete [ 5966.878485] Lustre: server umount lustre-OST0000 complete [ 5970.271345] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:31:46 (1776875506) [ 5972.729407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5972.909697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5974.288203] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5977.064718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5977.183516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5978.532391] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5980.272365] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5981.440345] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5996.706679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5996.768929] Lustre: Found index 0 for lustre-OST0000, updating log [ 5996.828995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5998.560277] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5998.887562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6000.252742] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6002.402901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6002.403869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6002.407870] Lustre: Skipped 2 previous similar messages [ 6007.500289] Lustre: server umount lustre-OST0000 complete [ 6023.136918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6023.139129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6023.142576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6030.305111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6030.307424] Lustre: Skipped 5 previous similar messages [ 6034.400140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6034.468641] Lustre: server umount lustre-MDT0000 complete [ 6035.425511] LustreError: 419285:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6035.433744] LustreError: 419285:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 6035.804274] LustreError: 419265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776875572 with bad export cookie 8042375554046005512 [ 6035.808966] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6035.810804] LustreError: 419265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6035.956477] Lustre: server umount lustre-MDT0001 complete [ 6039.046448] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:32:55 (1776875575) [ 6058.976123] LNet: 1 local NIs in recovery (showing 1): 192.168.204.137@tcp [ 6620.476430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6620.677096] LustreError: 425345:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6620.716497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6622.163097] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6625.435707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6627.048763] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6628.756931] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6630.222200] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6645.797253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6645.939918] Lustre: Found index 0 for lustre-OST0000, updating log [ 6648.033789] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6648.104397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6650.174443] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6652.195655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6652.203986] Lustre: Skipped 1 previous similar message [ 6652.207571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6653.221932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6657.306897] Lustre: server umount lustre-OST0000 complete [ 6659.553715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6659.554653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6659.560224] Lustre: Skipped 1 previous similar message [ 6659.561868] Lustre: Skipped 1 previous similar message [ 6663.649212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6663.651008] Lustre: Skipped 8 previous similar messages [ 6664.728938] Lustre: server umount lustre-MDT0000 complete [ 6665.697083] LustreError: 426056:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6665.702009] LustreError: 426056:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6665.904112] LustreError: 425327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776876202 with bad export cookie 8042375554046006569 [ 6665.906631] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6665.907905] LustreError: 425327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6666.020391] Lustre: server umount lustre-MDT0001 complete [ 6673.376309] LNet: 1 local NIs in recovery (showing 1): 192.168.204.137@tcp [ 6683.301349] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 6686.222360] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 6698.583438] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:43:54 (1776876234) [ 6706.109631] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 6707.258678] Key type lgssc unregistered [ 6707.404808] LNet: 431599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6707.409526] LNetError: 431599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6776.038462] LNet: Removed LNI 192.168.204.137@tcp [ 6776.335125] Key type .llcrypt unregistered [ 6776.336316] Key type ._llcrypt unregistered [ 6783.052089] Key type ._llcrypt registered [ 6783.053352] Key type .llcrypt registered [ 6783.089583] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 6783.460071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6783.468389] alg: No test for adler32 (adler32-zlib) [ 6784.363967] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 6784.458627] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 6786.048157] Key type lgssc registered [ 6786.382775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6788.030310] LNet: Added LNI 192.168.204.137@tcp100 [8/256/0/180] [ 6788.472451] LNet: Added LNI 192.168.204.137@tcp101 [8/256/0/180] [ 6788.801102] LNet: Added LNI 192.168.204.137@tcp102 [8/256/0/180] [ 6789.089930] LNet: Added LNI 192.168.204.137@tcp103 [8/256/0/180] [ 6789.389398] LNet: Added LNI 192.168.204.137@tcp104 [8/256/0/180] [ 6789.690480] LNet: Added LNI 192.168.204.137@tcp105 [8/256/0/180] [ 6790.059414] LNet: Added LNI 192.168.204.137@tcp106 [8/256/0/180] [ 6790.461620] LNet: Added LNI 192.168.204.137@tcp107 [8/256/0/180] [ 6790.770971] LNet: Added LNI 192.168.204.137@tcp108 [8/256/0/180] [ 6791.050775] LNet: Added LNI 192.168.204.137@tcp109 [8/256/0/180] [ 6791.387102] LNet: Added LNI 192.168.204.137@tcp110 [8/256/0/180] [ 6791.841793] LNet: Added LNI 192.168.204.137@tcp111 [8/256/0/180] [ 6792.205164] LNet: Added LNI 192.168.204.137@tcp112 [8/256/0/180] [ 6792.484344] LNet: Added LNI 192.168.204.137@tcp113 [8/256/0/180] [ 6792.790737] LNet: Added LNI 192.168.204.137@tcp114 [8/256/0/180] [ 6793.089046] LNet: Added LNI 192.168.204.137@tcp115 [8/256/0/180] [ 6793.375718] LNet: Added LNI 192.168.204.137@tcp116 [8/256/0/180] [ 6793.664863] LNet: Added LNI 192.168.204.137@tcp117 [8/256/0/180] [ 6793.951822] LNet: Added LNI 192.168.204.137@tcp118 [8/256/0/180] [ 6794.309193] LNet: Added LNI 192.168.204.137@tcp119 [8/256/0/180] [ 6794.683483] LNet: Added LNI 192.168.204.137@tcp120 [8/256/0/180] [ 6795.060843] LNet: Added LNI 192.168.204.137@tcp121 [8/256/0/180] [ 6795.379598] LNet: Added LNI 192.168.204.137@tcp122 [8/256/0/180] [ 6795.699465] LNet: Added LNI 192.168.204.137@tcp123 [8/256/0/180] [ 6796.023264] LNet: Added LNI 192.168.204.137@tcp124 [8/256/0/180] [ 6796.326712] LNet: Added LNI 192.168.204.137@tcp125 [8/256/0/180] [ 6796.623121] LNet: Added LNI 192.168.204.137@tcp126 [8/256/0/180] [ 6796.911532] LNet: Added LNI 192.168.204.137@tcp127 [8/256/0/180] [ 6797.195648] LNet: Added LNI 192.168.204.137@tcp128 [8/256/0/180] [ 6797.492323] LNet: Added LNI 192.168.204.137@tcp129 [8/256/0/180] [ 6797.777766] LNet: Added LNI 192.168.204.137@tcp130 [8/256/0/180] [ 6798.062269] LNet: Added LNI 192.168.204.137@tcp131 [8/256/0/180] [ 6798.350887] LNet: Added LNI 192.168.204.137@tcp132 [8/256/0/180] [ 6798.629954] LNet: Added LNI 192.168.204.137@tcp133 [8/256/0/180] [ 6798.912607] LNet: Added LNI 192.168.204.137@tcp134 [8/256/0/180] [ 6799.206573] LNet: Added LNI 192.168.204.137@tcp135 [8/256/0/180] [ 6801.364221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6801.370834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6802.419392] Lustre: 434580:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.137@tcp: can't add NID 192.168.204.137@tcp130: rc = -75 [ 6802.443644] LNetError: 434602:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.137 [ 6802.446765] LNetError: Protocol error connecting to 192.168.204.137@tcp112 at host 192.168.204.137:988: is it running a compatible version of Lustre? [ 6802.496792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6802.508449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6802.547096] Lustre: lustre-MDT0000: new disk, initializing [ 6802.579339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6802.587779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6803.491519] LNetError: 432046:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.137 [ 6803.495805] LNetError: Protocol error connecting to 192.168.204.137@tcp107 at host 192.168.204.137:988: is it running a compatible version of Lustre? [ 6803.987619] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6808.441300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6808.490896] Lustre: 434597:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6808.518521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6808.520583] Lustre: Skipped 1 previous similar message [ 6808.555964] Lustre: lustre-MDT0001: new disk, initializing [ 6808.567153] Lustre: 435382:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.137@tcp: can't add NID 192.168.204.137@tcp130: rc = -75 [ 6808.569964] Lustre: 435382:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6808.606702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6808.617372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6808.620776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6809.833062] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6812.898822] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6814.061146] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6816.199770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6816.307045] Lustre: lustre-OST0000: new disk, initializing [ 6816.309737] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6816.347413] Lustre: 436349:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.137@tcp: can't add NID 192.168.204.137@tcp130: rc = -75 [ 6816.354517] Lustre: 436349:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [ 6816.374374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6818.091688] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6818.925591] Lustre: 436783:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.137@tcp: can't add NID 192.168.204.137@tcp130: rc = -75 [ 6818.928303] Lustre: 436783:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6818.943480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6818.946530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6818.983489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6821.058840] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6824.419327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6824.419752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6824.424580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6824.426706] LustreError: Skipped 1 previous similar message [ 6824.429760] Lustre: Skipped 1 previous similar message [ 6829.322677] Lustre: server umount lustre-OST0000 complete [ 6834.144822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6834.147154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6834.150742] Lustre: Skipped 1 previous similar message [ 6834.152369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6836.638347] Lustre: server umount lustre-MDT0000 complete [ 6837.886833] LustreError: 435381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776876374 with bad export cookie 14352239005696517938 [ 6837.890768] LustreError: 435381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6837.892026] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6838.030164] Lustre: server umount lustre-MDT0001 complete [ 6841.768780] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 6842.806241] Key type lgssc unregistered [ 6842.849509] LNetError: 434602:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.137 [ 6842.853548] LNetError: Protocol error connecting to 192.168.204.137@tcp112 at host 192.168.204.137:988: is it running a compatible version of Lustre? [ 6842.973283] LNet: 438124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6842.978112] LNetError: 438124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6842.981975] LNet: Removed LNI 192.168.204.137@tcp [ 6844.000102] LNet: Removed LNI 192.168.204.137@tcp100 [ 6844.001547] LNetError: 438124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6844.003698] LNetError: 438124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6844.005995] LNet: Removed LNI 192.168.204.137@tcp101 [ 6845.024147] LNet: Removed LNI 192.168.204.137@tcp102 [ 6845.025680] LNetError: 438124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6845.027845] LNetError: 438124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6845.030121] LNet: Removed LNI 192.168.204.137@tcp103 [ 6846.048107] LNet: Removed LNI 192.168.204.137@tcp104 [ 6847.072145] LNet: Removed LNI 192.168.204.137@tcp105 [ 6847.074637] LNetError: 438124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6847.078955] LNetError: 438124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 6847.083624] LNet: Removed LNI 192.168.204.137@tcp106 [ 6848.096120] LNet: Removed LNI 192.168.204.137@tcp107 [ 6849.120174] LNet: Removed LNI 192.168.204.137@tcp108 [ 6850.144131] LNet: Removed LNI 192.168.204.137@tcp109 [ 6851.168126] LNet: Removed LNI 192.168.204.137@tcp110 [ 6851.169419] LNetError: 438124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6851.171591] LNetError: 438124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 6851.173854] LNet: Removed LNI 192.168.204.137@tcp111 [ 6852.192135] LNet: Removed LNI 192.168.204.137@tcp112 [ 6853.216114] LNet: Removed LNI 192.168.204.137@tcp113 [ 6854.240109] LNet: Removed LNI 192.168.204.137@tcp114 [ 6855.264132] LNet: Removed LNI 192.168.204.137@tcp115 [ 6856.288102] LNet: Removed LNI 192.168.204.137@tcp116 [ 6857.312120] LNet: Removed LNI 192.168.204.137@tcp117 [ 6857.313450] LNet: Removed LNI 192.168.204.137@tcp118 [ 6857.314687] LNet: Removed LNI 192.168.204.137@tcp119 [ 6857.315944] LNet: Removed LNI 192.168.204.137@tcp120 [ 6857.317079] LNet: Removed LNI 192.168.204.137@tcp121 [ 6857.318220] LNet: Removed LNI 192.168.204.137@tcp122 [ 6857.319323] LNet: Removed LNI 192.168.204.137@tcp123 [ 6857.320471] LNet: Removed LNI 192.168.204.137@tcp124 [ 6857.321634] LNet: Removed LNI 192.168.204.137@tcp125 [ 6857.322943] LNet: Removed LNI 192.168.204.137@tcp126 [ 6857.324155] LNet: Removed LNI 192.168.204.137@tcp127 [ 6857.325405] LNet: Removed LNI 192.168.204.137@tcp128 [ 6857.326705] LNet: Removed LNI 192.168.204.137@tcp129 [ 6857.327827] LNet: Removed LNI 192.168.204.137@tcp130 [ 6857.329034] LNet: Removed LNI 192.168.204.137@tcp131 [ 6857.330140] LNet: Removed LNI 192.168.204.137@tcp132 [ 6857.331346] LNet: Removed LNI 192.168.204.137@tcp133 [ 6857.332519] LNet: Removed LNI 192.168.204.137@tcp134 [ 6857.339446] LNet: Removed LNI 192.168.204.137@tcp135 [ 6857.606718] Key type .llcrypt unregistered [ 6857.607915] Key type ._llcrypt unregistered [ 6864.609505] Key type ._llcrypt registered [ 6864.610636] Key type .llcrypt registered [ 6864.645885] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 6870.720461] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 6871.180860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6871.196150] alg: No test for adler32 (adler32-zlib) [ 6872.043167] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 6872.120803] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 6873.704188] Key type lgssc registered [ 6874.070802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6884.751466] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:47:00 (1776876420) [ 6890.290322] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 6891.311542] Key type lgssc unregistered [ 6891.442520] LNet: 442130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6891.447165] LNetError: 442130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6891.455479] LNet: Removed LNI 192.168.204.137@tcp [ 6891.775479] Key type .llcrypt unregistered [ 6891.776655] Key type ._llcrypt unregistered [ 6897.798157] Key type ._llcrypt registered [ 6897.799139] Key type .llcrypt registered [ 6897.834059] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 6898.333511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6898.343970] alg: No test for adler32 (adler32-zlib) [ 6899.199962] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 6899.287064] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 6900.864186] Key type lgssc registered [ 6901.219656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6904.123080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6904.128149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6905.213541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6905.225352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6905.263359] Lustre: lustre-MDT0000: new disk, initializing [ 6905.302298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6905.308811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6906.739464] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6910.942066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6910.990804] Lustre: 443301:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6911.020730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6911.024170] Lustre: Skipped 1 previous similar message [ 6911.069169] Lustre: lustre-MDT0001: new disk, initializing [ 6911.108767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6911.122476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6911.126096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6912.673910] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6915.856696] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6917.169700] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6919.510484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6919.627949] Lustre: lustre-OST0000: new disk, initializing [ 6919.629734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6919.662692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6921.209219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6921.212381] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6921.246163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6921.600091] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6924.855279] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6926.128552] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing lnet_if_list [ 6928.039105] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 6928.054367] LNet: Added LNI 192.168.204.137@tcp10 [8/256/0/180] [ 6929.961887] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 6929.975081] LNet: Added LNI 192.168.204.137@tcp11 [8/256/0/180] [ 6931.249389] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 6931.259896] LNet: Added LNI 192.168.204.137@tcp12 [8/256/0/180] [ 6932.678487] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 6932.689667] LNet: Added LNI 192.168.204.137@tcp13 [8/256/0/180] [ 6934.031253] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 6934.042912] LNet: Added LNI 192.168.204.137@tcp14 [8/256/0/180] [ 6935.365281] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 6935.376699] LNet: Added LNI 192.168.204.137@tcp15 [8/256/0/180] [ 6936.874351] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 6936.885239] LNet: Added LNI 192.168.204.137@tcp16 [8/256/0/180] [ 6938.320778] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 6938.331937] LNet: Added LNI 192.168.204.137@tcp17 [8/256/0/180] [ 6939.746582] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 6939.757631] LNet: Added LNI 192.168.204.137@tcp18 [8/256/0/180] [ 6941.381142] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 6941.392262] LNet: Added LNI 192.168.204.137@tcp19 [8/256/0/180] [ 6943.206776] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 6943.222494] LNet: Added LNI 192.168.204.137@tcp20 [8/256/0/180] [ 6944.925436] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 6944.935233] LNet: Added LNI 192.168.204.137@tcp21 [8/256/0/180] [ 6946.496156] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 6946.506976] LNet: Added LNI 192.168.204.137@tcp22 [8/256/0/180] [ 6948.310589] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 6948.323460] LNet: Added LNI 192.168.204.137@tcp23 [8/256/0/180] [ 6950.132304] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 6950.144089] LNet: Added LNI 192.168.204.137@tcp24 [8/256/0/180] [ 6951.896911] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 6951.910258] LNet: Added LNI 192.168.204.137@tcp25 [8/256/0/180] [ 6953.623629] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 6953.633785] LNet: Added LNI 192.168.204.137@tcp26 [8/256/0/180] [ 6955.457387] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 6955.469476] LNet: Added LNI 192.168.204.137@tcp27 [8/256/0/180] [ 6957.328082] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 6957.338792] LNet: Added LNI 192.168.204.137@tcp28 [8/256/0/180] [ 6959.113322] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 6959.128283] LNet: Added LNI 192.168.204.137@tcp29 [8/256/0/180] [ 6960.943538] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 6960.953703] LNet: Added LNI 192.168.204.137@tcp30 [8/256/0/180] [ 6962.859094] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 6962.875470] LNet: Added LNI 192.168.204.137@tcp31 [8/256/0/180] [ 6964.886267] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 6964.898337] LNet: Added LNI 192.168.204.137@tcp32 [8/256/0/180] [ 6967.087708] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 6967.100731] LNet: Added LNI 192.168.204.137@tcp33 [8/256/0/180] [ 6969.208387] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 6969.217743] LNet: Added LNI 192.168.204.137@tcp34 [8/256/0/180] [ 6971.255700] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 6971.272597] LNet: Added LNI 192.168.204.137@tcp35 [8/256/0/180] [ 6973.991925] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 6974.006208] LNet: Added LNI 192.168.204.137@tcp36 [8/256/0/180] [ 6976.284825] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 6976.307180] LNet: Added LNI 192.168.204.137@tcp37 [8/256/0/180] [ 6979.086877] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 6979.103601] LNet: Added LNI 192.168.204.137@tcp38 [8/256/0/180] [ 6981.352325] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 6981.365233] LNet: Added LNI 192.168.204.137@tcp39 [8/256/0/180] [ 6983.979622] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 6983.989550] LNet: Added LNI 192.168.204.137@tcp40 [8/256/0/180] [ 6986.571886] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 6986.584301] LNet: Added LNI 192.168.204.137@tcp41 [8/256/0/180] [ 6989.232669] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 6989.246444] LNet: Added LNI 192.168.204.137@tcp42 [8/256/0/180] [ 6992.356758] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 6992.366518] LNet: Added LNI 192.168.204.137@tcp43 [8/256/0/180] [ 6995.149544] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 6995.164649] LNet: Added LNI 192.168.204.137@tcp44 [8/256/0/180] [ 6998.665992] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 6998.687187] LNet: Added LNI 192.168.204.137@tcp45 [8/256/0/180] [ 7001.678097] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7001.690042] LNet: Added LNI 192.168.204.137@tcp46 [8/256/0/180] [ 7003.903016] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7003.913577] LNet: Added LNI 192.168.204.137@tcp47 [8/256/0/180] [ 7006.274170] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7006.282884] LNet: Added LNI 192.168.204.137@tcp48 [8/256/0/180] [ 7009.446255] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7009.455920] LNet: Added LNI 192.168.204.137@tcp49 [8/256/0/180] [ 7012.741698] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7012.758711] LNet: Added LNI 192.168.204.137@tcp50 [8/256/0/180] [ 7023.589668] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7023.591318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7023.599233] Lustre: Skipped 2 previous similar messages [ 7023.603236] Lustre: Skipped 1 previous similar message [ 7027.768799] Lustre: server umount lustre-MDT0000 complete [ 7029.634878] LustreError: 443295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776876566 with bad export cookie 7343419974754677653 [ 7029.641897] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7029.642804] LustreError: 443295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7029.758952] Lustre: server umount lustre-MDT0001 complete [ 7038.727279] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 7042.122334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7050.208156] Lustre: 442575:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776876570/real 1776876570] req@000000007b842440 x1863189984789760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776876586 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7050.224723] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7051.232180] Lustre: 442577:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776876571/real 1776876571] req@00000000b51fbee3 x1863189984789888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776876587 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7054.369487] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x65e9118f57c2bb8a [ 7054.375201] Lustre: MGC192.168.204.137@tcp: Connection restored to 0@lo (at 0@lo) [ 7054.540589] LustreError: 445034:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7054.549221] LustreError: 445034:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 7054.568590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7055.392146] Lustre: 442575:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776876576/real 1776876576] req@00000000fa9da79c x1863189984790272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776876592 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7055.585079] LustreError: 445035:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7056.275642] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7059.619086] LustreError: 445034:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7060.600922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7060.744585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7061.792196] Lustre: 442576:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776876581/real 1776876581] req@00000000887b9aed x1863189984790400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776876597 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7061.807921] Lustre: 442576:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7062.481980] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7065.136871] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7065.824112] Lustre: 442577:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776876586/real 1776876586] req@00000000f3d85f60 x1863189984790528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776876602 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7066.090243] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7066.797784] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7076.327202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7076.328567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7076.335870] Lustre: Skipped 1 previous similar message [ 7076.336659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7076.340249] Lustre: Skipped 2 previous similar messages [ 7081.445310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7081.449689] Lustre: Skipped 1 previous similar message [ 7082.228606] Lustre: server umount lustre-OST0000 complete [ 7086.562843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7086.563528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7086.564823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7086.580795] Lustre: Skipped 1 previous similar message [ 7090.332761] Lustre: server umount lustre-MDT0000 complete [ 7091.681812] LustreError: 468390:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7091.692178] LustreError: 468390:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 7092.049483] LustreError: 443293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776876628 with bad export cookie 7343419974754679690 [ 7092.056107] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7092.056234] LustreError: 443293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7092.202725] Lustre: server umount lustre-MDT0001 complete [ 7097.429029] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 7099.183615] Key type lgssc unregistered [ 7099.428799] LNet: 470919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7099.433164] LNetError: 470919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7099.436085] LNet: Removed LNI 192.168.204.137@tcp [ 7099.437983] LNet: Removed LNI 192.168.204.137@tcp10 [ 7099.440407] LNet: Removed LNI 192.168.204.137@tcp11 [ 7099.442137] LNet: Removed LNI 192.168.204.137@tcp12 [ 7099.443956] LNet: Removed LNI 192.168.204.137@tcp13 [ 7099.445976] LNet: Removed LNI 192.168.204.137@tcp14 [ 7099.448205] LNet: Removed LNI 192.168.204.137@tcp15 [ 7099.450150] LNet: Removed LNI 192.168.204.137@tcp16 [ 7099.451877] LNet: Removed LNI 192.168.204.137@tcp17 [ 7099.454038] LNet: Removed LNI 192.168.204.137@tcp18 [ 7099.456276] LNet: Removed LNI 192.168.204.137@tcp19 [ 7099.458427] LNet: Removed LNI 192.168.204.137@tcp20 [ 7099.460139] LNet: Removed LNI 192.168.204.137@tcp21 [ 7099.461634] LNet: Removed LNI 192.168.204.137@tcp22 [ 7099.463370] LNet: Removed LNI 192.168.204.137@tcp23 [ 7099.464945] LNet: Removed LNI 192.168.204.137@tcp24 [ 7099.466341] LNet: Removed LNI 192.168.204.137@tcp25 [ 7099.468082] LNet: Removed LNI 192.168.204.137@tcp26 [ 7099.469562] LNet: Removed LNI 192.168.204.137@tcp27 [ 7099.470890] LNet: Removed LNI 192.168.204.137@tcp28 [ 7099.472297] LNet: Removed LNI 192.168.204.137@tcp29 [ 7099.473825] LNet: Removed LNI 192.168.204.137@tcp30 [ 7099.475450] LNet: Removed LNI 192.168.204.137@tcp31 [ 7099.477165] LNet: Removed LNI 192.168.204.137@tcp32 [ 7099.478822] LNet: Removed LNI 192.168.204.137@tcp33 [ 7099.480156] LNet: Removed LNI 192.168.204.137@tcp34 [ 7099.481369] LNet: Removed LNI 192.168.204.137@tcp35 [ 7099.482531] LNet: Removed LNI 192.168.204.137@tcp36 [ 7099.483944] LNet: Removed LNI 192.168.204.137@tcp37 [ 7099.485495] LNet: Removed LNI 192.168.204.137@tcp38 [ 7099.486802] LNet: Removed LNI 192.168.204.137@tcp39 [ 7099.488059] LNet: Removed LNI 192.168.204.137@tcp40 [ 7099.489235] LNet: Removed LNI 192.168.204.137@tcp41 [ 7099.490585] LNet: Removed LNI 192.168.204.137@tcp42 [ 7099.491979] LNet: Removed LNI 192.168.204.137@tcp43 [ 7099.493341] LNet: Removed LNI 192.168.204.137@tcp44 [ 7099.494646] LNet: Removed LNI 192.168.204.137@tcp45 [ 7099.495761] LNet: Removed LNI 192.168.204.137@tcp46 [ 7099.496872] LNet: Removed LNI 192.168.204.137@tcp47 [ 7099.498155] LNet: Removed LNI 192.168.204.137@tcp48 [ 7099.499202] LNet: Removed LNI 192.168.204.137@tcp49 [ 7099.506584] LNet: Removed LNI 192.168.204.137@tcp50 [ 7099.956980] Key type .llcrypt unregistered [ 7099.960072] Key type ._llcrypt unregistered [ 7109.253795] Key type ._llcrypt registered [ 7109.255304] Key type .llcrypt registered [ 7109.315002] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 7116.963846] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 7117.395913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7117.403753] alg: No test for adler32 (adler32-zlib) [ 7118.283166] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 7118.384534] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 7119.976245] Key type lgssc registered [ 7120.526250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7133.947271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7133.952695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7135.062096] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7135.077993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7135.129128] Lustre: lustre-MDT0000: new disk, initializing [ 7135.151367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7135.157123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7136.380290] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7140.315160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7140.357796] Lustre: 474057:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7140.378047] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7140.379808] Lustre: Skipped 1 previous similar message [ 7140.418510] Lustre: lustre-MDT0001: new disk, initializing [ 7140.449696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7140.462937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7140.466069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7141.630142] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7144.556777] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7145.761306] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7148.298971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7148.377578] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7148.412136] Lustre: lustre-OST0000: new disk, initializing [ 7148.414561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7148.446426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7150.426865] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7151.741099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7151.744520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7151.775878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7153.531241] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7157.218239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7157.220648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7157.223371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7157.223800] Lustre: Skipped 1 previous similar message [ 7161.864393] Lustre: server umount lustre-OST0000 complete [ 7165.920911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7165.923804] LustreError: Skipped 1 previous similar message [ 7165.925457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7165.930151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7165.932566] Lustre: Skipped 1 previous similar message [ 7167.457153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7167.457634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7169.171649] Lustre: server umount lustre-MDT0000 complete [ 7170.481044] LustreError: 474049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776876707 with bad export cookie 16002277047560799152 [ 7170.484552] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7170.485050] LustreError: 474049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7170.592789] Lustre: server umount lustre-MDT0001 complete [ 7174.436184] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 7175.801793] Key type lgssc unregistered [ 7175.929415] LNet: 477619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7175.932534] LNetError: 477619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7175.942416] LNet: Removed LNI 192.168.204.137@tcp [ 7176.266651] Key type .llcrypt unregistered [ 7176.267956] Key type ._llcrypt unregistered [ 7184.411363] Key type ._llcrypt registered [ 7184.412701] Key type .llcrypt registered [ 7184.448185] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 7184.766954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7184.813079] alg: No test for adler32 (adler32-zlib) [ 7185.665004] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 7185.750471] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 7187.328121] Key type lgssc registered [ 7187.669839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7190.954595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7190.964940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7192.134583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7193.547471] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7196.840379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7196.969480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7198.431254] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7200.762436] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7201.891528] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7204.270541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7204.430614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7206.343387] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7208.199008] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7222.751147] loop0: detected capacity change from 0 to 204800000 [ 7222.813772] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7222.818313] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7222.823882] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7225.278570] loop0: detected capacity change from 0 to 204800000 [ 7225.364428] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7225.372328] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7225.378503] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7227.412052] loop0: detected capacity change from 0 to 204800000 [ 7229.708796] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7229.765843] Lustre: fs15246-OST0000: new disk, initializing [ 7229.768690] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7229.790790] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7232.088862] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7236.346826] loop1: detected capacity change from 0 to 204800000 [ 7238.171798] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7238.210838] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7238.236538] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7238.282097] Lustre: fs15246-MDT0000: new disk, initializing [ 7238.339724] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7238.353051] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7238.417919] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7239.457678] Lustre: 483224:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000a3e31a3f x1863190285681536/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1776876792 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7239.857824] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7248.544325] LustreError: 483224:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7248.547738] LustreError: 483224:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 7248.550592] LustreError: 483224:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 7252.453881] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7252.463936] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7253.591561] Lustre: server umount fs15246-MDT0000 complete [ 7256.660288] Lustre: server umount fs15246-OST0000 complete [ 7262.697368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7262.698386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7262.706220] Lustre: Skipped 1 previous similar message [ 7262.711708] Lustre: Skipped 1 previous similar message [ 7266.281383] Lustre: server umount lustre-OST0000 complete [ 7268.833528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7268.838971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7268.847956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7272.930474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7272.932004] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.234652] Lustre: server umount lustre-MDT0000 complete [ 7275.714673] LustreError: 478830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776876812 with bad export cookie 12333046147663440549 [ 7275.718101] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7275.718610] LustreError: 478830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7275.831503] Lustre: server umount lustre-MDT0001 complete [ 7280.193877] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 7281.570461] Key type lgssc unregistered [ 7281.708496] LNet: 486283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7281.711760] LNetError: 486283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7281.719504] LNet: Removed LNI 192.168.204.137@tcp [ 7282.144959] Key type .llcrypt unregistered [ 7282.146501] Key type ._llcrypt unregistered [ 7289.101684] Key type ._llcrypt registered [ 7289.104479] Key type .llcrypt registered [ 7289.138641] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 7289.466095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7289.498513] alg: No test for adler32 (adler32-zlib) [ 7290.367696] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 7290.450533] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 7292.024102] Key type lgssc registered [ 7292.344181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7303.778639] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 7306.531323] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 7317.453491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7317.459202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7318.543333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7318.554655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7318.593787] Lustre: lustre-MDT0000: new disk, initializing [ 7318.615960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7318.624121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7319.809218] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7323.719588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7323.759447] Lustre: 490664:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7323.781289] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7323.783167] Lustre: Skipped 1 previous similar message [ 7323.816721] Lustre: lustre-MDT0001: new disk, initializing [ 7323.847528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7323.859115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7323.864561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7325.122260] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7328.141916] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7329.354645] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7331.647436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7331.748534] Lustre: lustre-OST0000: new disk, initializing [ 7331.750924] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7331.778908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7333.174977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7333.178235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7333.212972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7333.671660] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7336.795804] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7338.158625] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7338.243885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7339.475066] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7339.546784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7342.230366] Lustre: server umount lustre-OST0000 complete [ 7343.586541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7343.587426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7343.590382] Lustre: Skipped 1 previous similar message [ 7348.706056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7348.707799] Lustre: Skipped 2 previous similar messages [ 7353.825327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7353.827882] Lustre: Skipped 1 previous similar message [ 7353.858331] Lustre: server umount lustre-MDT0000 complete [ 7355.102902] LustreError: 490657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776876891 with bad export cookie 8359168600173287634 [ 7355.105502] LustreError: MGC192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7355.106723] LustreError: 490657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7364.632377] Lustre: server umount lustre-MDT0001 complete [ 7372.577925] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 7376.340527] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 7387.394562] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7234 sec ========== 12:55:23 (1776876923) [ 7387.952853] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:55:23 (1776876923) === [ 7389.161529] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:55:25 (1776876925) === [ 7403.811480] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 7404.845631] Key type lgssc unregistered [ 7404.978483] LNet: 498838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7404.980891] LNetError: 498838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7404.993419] LNet: Removed LNI 192.168.204.137@tcp [ 7405.283900] Key type .llcrypt unregistered [ 7405.285900] Key type ._llcrypt unregistered