[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 505917541 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002317] x2apic enabled [ 0.004004] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010122] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013032] SELinux: Initializing. [ 0.014092] *** VALIDATE selinux *** [ 0.022801] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027683] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029133] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030131] *** VALIDATE tmpfs *** [ 0.032007] *** VALIDATE proc *** [ 0.033236] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035263] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036172] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038034] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042597] debug: unmapping init [mem 0xffffffffa5a59000-0xffffffffa5a60fff] [ 0.045154] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046626] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049009] ... generic registers: 4 [ 0.050011] ... value mask: 0000ffffffffffff [ 0.051011] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053011] ... event mask: 000000070000000f [ 0.054343] rcu: Hierarchical SRCU implementation. [ 0.056589] smp: Bringing up secondary CPUs ... [ 0.057602] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.065095] smp: Brought up 1 node, 4 CPUs [ 0.067012] smpboot: Max logical packages: 1 [ 0.068013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.124337] node 0 deferred pages initialised in 49ms [ 0.128657] devtmpfs: initialized [ 0.129225] x86/mm: Memory block size: 128MB [ 0.132289] gcov: version magic: 0x41383552 [ 0.134267] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.135299] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.136230] pinctrl core: initialized pinctrl subsystem [ 0.137213] [ 0.138009] ************************************************************* [ 0.139015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.140015] ** ** [ 0.141019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.142027] ** ** [ 0.143015] ** This means that this kernel is built to expose internal ** [ 0.144018] ** IOMMU data structures, which may compromise security on ** [ 0.145017] ** your system. ** [ 0.146018] ** ** [ 0.147014] ** If you see this message and you are not debugging the ** [ 0.148014] ** kernel, report this immediately to your vendor! ** [ 0.149012] ** ** [ 0.150013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151011] ************************************************************* [ 0.152952] NET: Registered protocol family 16 [ 0.153579] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.154238] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.155083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.157312] cpuidle: using governor menu [ 0.159052] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.160683] PCI: Using configuration type 1 for base access [ 0.161368] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.176073] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.178016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.181219] cryptd: max_cpu_qlen set to 1000 [ 0.182294] ACPI: Added _OSI(Module Device) [ 0.184014] ACPI: Added _OSI(Processor Device) [ 0.186212] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.189014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.196936] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.206785] ACPI: Interpreter enabled [ 0.207115] ACPI: PM: (supports S0 S3 S4 S5) [ 0.208037] ACPI: Using IOAPIC for interrupt routing [ 0.209173] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.210622] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.225684] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.227022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228106] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.231380] acpiphp: Slot [2] registered [ 0.232193] acpiphp: Slot [5] registered [ 0.233229] acpiphp: Slot [6] registered [ 0.234426] acpiphp: Slot [7] registered [ 0.235369] acpiphp: Slot [8] registered [ 0.236171] acpiphp: Slot [9] registered [ 0.237090] acpiphp: Slot [10] registered [ 0.238232] acpiphp: Slot [3] registered [ 0.239190] acpiphp: Slot [4] registered [ 0.240192] acpiphp: Slot [11] registered [ 0.241125] acpiphp: Slot [12] registered [ 0.242099] acpiphp: Slot [13] registered [ 0.243110] acpiphp: Slot [14] registered [ 0.244109] acpiphp: Slot [15] registered [ 0.245080] acpiphp: Slot [16] registered [ 0.246242] acpiphp: Slot [17] registered [ 0.247080] acpiphp: Slot [18] registered [ 0.248108] acpiphp: Slot [19] registered [ 0.249113] acpiphp: Slot [20] registered [ 0.250175] acpiphp: Slot [21] registered [ 0.251475] acpiphp: Slot [22] registered [ 0.253125] acpiphp: Slot [23] registered [ 0.254537] acpiphp: Slot [24] registered [ 0.255327] acpiphp: Slot [25] registered [ 0.256109] acpiphp: Slot [26] registered [ 0.257218] acpiphp: Slot [27] registered [ 0.258134] acpiphp: Slot [28] registered [ 0.259231] acpiphp: Slot [29] registered [ 0.260147] acpiphp: Slot [30] registered [ 0.261104] acpiphp: Slot [31] registered [ 0.262040] PCI host bridge to bus 0000:00 [ 0.263080] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.264048] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.266177] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.270023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.273018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.276043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.278170] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.281092] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.284409] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.304014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.311063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.316019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.318017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.321015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.324192] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.327831] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.330081] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.333851] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.338000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.352014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.358013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.365832] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.384017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.430015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.510062] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.561818] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.573016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.582078] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.609014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.638479] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.646013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.657015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.683015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.701388] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.710015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.717016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.744028] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.754405] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.761026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.779028] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.051022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.062782] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.073016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.081000] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.089017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.107000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.109361] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.111432] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.114426] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.115222] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.118185] iommu: Default domain type: Passthrough [ 1.119670] SCSI subsystem initialized [ 1.120000] ACPI: bus type USB registered [ 1.120000] usbcore: registered new interface driver usbfs [ 1.120000] usbcore: registered new interface driver hub [ 1.120000] usbcore: registered new device driver usb [ 1.120000] pps_core: LinuxPPS API ver. 1 registered [ 1.127066] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.129055] PTP clock support registered [ 1.131116] EDAC MC: Ver: 3.0.0 [ 1.137140] PCI: Using ACPI for IRQ routing [ 1.140201] NetLabel: Initializing [ 1.141013] NetLabel: domain hash size = 128 [ 1.143012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.145182] NetLabel: unlabeled traffic allowed by default [ 1.147325] vgaarb: loaded [ 1.149607] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.151010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.158136] clocksource: Switched to clocksource kvm-clock [ 1.473587] VFS: Disk quotas dquot_6.6.0 [ 1.518464] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.612536] *** VALIDATE ramfs *** [ 1.614542] *** VALIDATE hugetlbfs *** [ 1.617094] pnp: PnP ACPI init [ 1.620331] pnp: PnP ACPI: found 6 devices [ 1.653049] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.657363] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.660154] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.667654] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.676380] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.704767] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.714968] NET: Registered protocol family 2 [ 1.717903] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.729261] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.733170] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.748524] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.754388] TCP: Hash tables configured (established 65536 bind 65536) [ 1.761542] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.765761] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.769491] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.773115] NET: Registered protocol family 1 [ 1.777740] RPC: Registered named UNIX socket transport module. [ 1.779847] RPC: Registered udp transport module. [ 1.781402] RPC: Registered tcp transport module. [ 1.782861] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.784817] NET: Registered protocol family 44 [ 1.786351] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.788225] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.790105] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.792259] PCI: CLS 0 bytes, default 64 [ 1.793869] Unpacking initramfs... [ 3.616298] debug: unmapping init [mem 0xffff9407fcc54000-0xffff9407fffbffff] [ 3.627260] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.631101] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.635680] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.538229] Initialise system trusted keyrings [ 4.556108] Key type blacklist registered [ 4.559671] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.575487] zbud: loaded [ 4.580924] *** VALIDATE nfs *** [ 4.582906] *** VALIDATE nfs4 *** [ 4.587094] pstore: using deflate compression [ 4.617086] Platform Keyring initialized [ 4.916793] NET: Registered protocol family 38 [ 4.923155] Key type asymmetric registered [ 4.926024] Asymmetric key parser 'x509' registered [ 4.930071] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.935226] io scheduler mq-deadline registered [ 4.938569] io scheduler kyber registered [ 4.941695] io scheduler bfq registered [ 4.944098] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.948975] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.952786] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.956715] ACPI: Power Button [PWRF] [ 4.964543] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.973740] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.026739] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.042369] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.087590] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.124399] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.219453] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.224933] Non-volatile memory driver v1.3 [ 5.226890] Linux agpgart interface v0.103 [ 5.383020] virtio_blk virtio1: [vda] 145960 512-byte logical blocks (74.7 MB/71.3 MiB) [ 5.385971] vda: detected capacity change from 0 to 74731520 [ 6.303162] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.589711] vdb: detected capacity change from 0 to 1073741824 [ 6.650904] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.654947] vdc: detected capacity change from 0 to 2621440000 [ 6.780280] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.785497] vdd: detected capacity change from 0 to 2621440000 [ 6.802694] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.806723] vde: detected capacity change from 0 to 4294967296 [ 6.898678] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.902323] vdf: detected capacity change from 0 to 4294967296 [ 6.916105] libphy: Fixed MDIO Bus: probed [ 6.921792] usbcore: registered new interface driver usbserial_generic [ 6.924719] usbserial: USB Serial support registered for generic [ 6.928245] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.973675] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.975981] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.979289] mousedev: PS/2 mouse device common for all mice [ 6.983136] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.983423] rtc_cmos 00:05: RTC can wake from S4 [ 6.993651] rtc_cmos 00:05: registered as rtc0 [ 6.995868] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.999578] intel_pstate: CPU model not supported [ 7.001896] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.005727] hid: raw HID events driver (C) Jiri Kosina [ 7.014128] usbcore: registered new interface driver usbhid [ 7.016509] usbhid: USB HID core driver [ 7.018389] drop_monitor: Initializing network drop monitor service [ 7.021502] Initializing XFRM netlink socket [ 7.023838] NET: Registered protocol family 10 [ 7.027226] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.031901] Segment Routing with IPv6 [ 7.033969] NET: Registered protocol family 17 [ 7.036429] mpls_gso: MPLS GSO support [ 7.042255] RAS: Correctable Errors collector initialized. [ 7.044877] AVX version of gcm_enc/dec engaged. [ 7.046883] AES CTR mode by8 optimization enabled [ 7.133789] sched_clock: Marking stable (7133710466, 0)->(8595685170, -1461974704) [ 7.151901] registered taskstats version 1 [ 7.201656] Loading compiled-in X.509 certificates [ 7.203612] zswap: loaded using pool lzo/zbud [ 7.224522] Key type big_key registered [ 7.235621] Key type encrypted registered [ 7.237201] ima: No TPM chip found, activating TPM-bypass! [ 7.238896] ima: Allocated hash algorithm: sha1 [ 7.240420] ima: No architecture policies found [ 7.241842] evm: Initialising EVM extended attributes: [ 7.243417] evm: security.selinux [ 7.246107] evm: security.ima [ 7.248926] evm: security.capability [ 7.250787] evm: HMAC attrs: 0x1 [ 7.253381] rtc_cmos 00:05: setting system clock to 2026-09-08 21:28:49 UTC (1788902929) [ 7.264362] debug: unmapping init [mem 0xffffffffa6a03000-0xffffffffa6bfffff] [ 7.267919] debug: unmapping init [mem 0xffffffffa5782000-0xffffffffa5a58fff] [ 7.277140] Write protecting the kernel read-only data: 28672k [ 7.281044] debug: unmapping init [mem 0xffffffffa3e03000-0xffffffffa3ffffff] [ 7.283587] debug: unmapping init [mem 0xffffffffa4714000-0xffffffffa47fffff] [ 7.429246] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.440403] systemd[1]: Detected virtualization kvm. [ 7.442520] systemd[1]: Detected architecture x86-64. [ 7.444559] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.560542] systemd[1]: No hostname configured. [ 7.563012] systemd[1]: Set hostname to . [ 7.566660] random: systemd: uninitialized urandom read (16 bytes read) [ 7.574679] systemd[1]: Initializing machine ID from random generator. [ 8.239698] random: ln: uninitialized urandom read (6 bytes read) [ 8.394983] random: systemd: uninitialized urandom read (16 bytes read) [ 8.399195] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 8.406514] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 8.412585] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.271569] device-mapper: uevent: version 1.0.3 [ 10.275192] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 12.217201] virtio_net virtio0 ens2: renamed from eth0 [ 12.273218] random: fast init done [ 12.916469] scsi host0: ata_piix [ 13.042967] scsi host1: ata_piix [ 13.044751] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.047679] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.767545] random: crng init done [ 17.773310] random: 7 urandom warning(s) missed due to ratelimiting [ 18.852858] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 22.506087] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.948139] printk: systemd: 26 output lines suppressed due to ratelimiting [ 28.357444] SELinux: Disabled at runtime. [ 28.540445] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 28.571305] systemd[1]: Detected virtualization kvm. [ 28.574881] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 34.536600] systemd[1]: initrd-switch-root.service: Succeeded. [ 34.575050] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 34.789900] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 34.816104] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 34.822677] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 34.886379] systemd[1]: Starting Journal Service... Starting Journal Service... [ 34.942254] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ 34.982617] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ 35.034112] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ 35.085221] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ 35.347590] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 37.900561] hrtimer: interrupt took 4801165 ns [ 37.950856] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 39.613692] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 39.932825] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 41.310929] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 41.867641] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (10s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ *] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (14s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [*** ] A start job is running for Configur…only root support (16s / no limit) [** ] A start job is running for Configur…only root support (16s / no limit) [* ] A start job is running for Configur…only root support (17s / no limit) [** ] A start job is running for Configur…only root support (17s / no limit) [*** ] A start job is running for Configur…only root support (18s / no limit) [ *** ] A start job is running for Configur…only root support (18s / no limit)[ 53.293709] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (19s / no limit) [ ***] A start job is running for Configur…only root support (19s / no limit)[ 54.396975] NFS: Registering the id_resolver key type [ 54.399758] Key type id_resolver registered [ 54.401939] Key type id_legacy registered [ **] A start job is running for Configur…only root support (20s / no limit) [ *] A start job is running for Configur…only root support (20s / no limit) [ **] A start job is running for Configur…only root support (21s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg324-server login: [ 108.659525] libcfs: loading out-of-tree module taints kernel. [ 108.728936] Key type ._llcrypt registered [ 108.734475] Key type .llcrypt registered [ 108.890116] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 130.454166] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 131.654592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 131.669478] alg: No test for adler32 (adler32-zlib) [ 133.042179] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 133.759163] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 135.487169] Key type lgssc registered [ 137.038495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 154.568615] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 194.053798] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 207.165848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 207.221909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.542412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 208.598215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 208.741637] Lustre: lustre-MDT0000: new disk, initializing [ 208.896335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.912579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 213.179299] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.918437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 227.069982] Lustre: 6520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 227.104719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 227.116990] Lustre: Skipped 1 previous similar message [ 227.223608] Lustre: lustre-MDT0001: new disk, initializing [ 227.288445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 227.309481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 227.321146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 231.608496] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.465919] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 245.995855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 246.314225] Lustre: lustre-OST0000: new disk, initializing [ 246.317656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 246.323213] Lustre: 8459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 246.391288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 246.948359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 246.971354] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 247.056416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 252.398408] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 267.105887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 267.306568] Lustre: lustre-OST0001: new disk, initializing [ 267.312288] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 267.319730] Lustre: 9533:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 267.392393] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 273.621104] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.490178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 275.503972] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 275.569369] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 284.431300] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 289.724957] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 296.316399] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing check_logdir /tmp/testlogs/ [ 300.974923] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing yml_node [ 305.023872] Lustre: DEBUG MARKER: Client: 2.17.55.26 [ 307.500822] Lustre: DEBUG MARKER: MDS: 2.17.55.26 [ 310.524301] Lustre: DEBUG MARKER: OSS: 2.17.55.26 [ 312.692299] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 17:33:52 EDT 2026 [ 328.242869] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 329.829416] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 347.107244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.110630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.120615] Lustre: Skipped 3 previous similar messages [ 347.127822] Lustre: Skipped 3 previous similar messages [ 351.947321] Lustre: server umount lustre-MDT0000 complete [ 357.346276] LustreError: 6531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.374687] LustreError: 6531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 357.929781] LustreError: 6556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903280 with bad export cookie 8004714741722541127 [ 357.937772] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 357.998527] LustreError: 6556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 362.465546] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.472698] LustreError: 9283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 362.480861] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 362.484146] Lustre: Skipped 1 previous similar message [ 362.536432] LustreError: 9283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 364.505870] Lustre: server umount lustre-MDT0001 complete [ 370.086409] Lustre: server umount lustre-OST0000 complete [ 375.346881] Lustre: server umount lustre-OST0001 complete [ 382.657178] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 391.880619] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 428.630720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.805938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 428.822437] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 428.877421] Lustre: lustre-MDT0000: new disk, initializing [ 428.947944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.962895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 433.995979] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 446.456822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.610316] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 446.710772] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 446.715197] Lustre: Skipped 1 previous similar message [ 446.830858] Lustre: lustre-MDT0001: new disk, initializing [ 447.057738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 447.159445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 447.201587] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 452.759584] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 464.580441] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 471.301989] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 477.910426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 478.403800] Lustre: lustre-OST0000: new disk, initializing [ 478.409068] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 478.416330] Lustre: 17053:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 478.547648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.861037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 479.878772] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 480.056826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 484.133382] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 492.422647] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 495.902904] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 496.067978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 499.678613] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 499.823868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 505.832276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 505.832898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.846401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 505.848251] Lustre: Skipped 1 previous similar message [ 505.866244] Lustre: Skipped 2 previous similar messages [ 507.423612] Lustre: server umount lustre-OST0000 complete [ 510.944885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.952082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 510.957644] Lustre: Skipped 1 previous similar message [ 516.065046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 516.069888] Lustre: Skipped 2 previous similar messages [ 516.294483] Lustre: server umount lustre-MDT0000 complete [ 518.556232] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903440 with bad export cookie 8004714741722542660 [ 518.557118] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 518.564457] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 518.794535] Lustre: server umount lustre-MDT0001 complete [ 521.635435] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 17:37:22 (1788903442) [ 523.433823] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 524.845569] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 17:37:25 (1788903445) [ 530.432652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.899858] LustreError: 19459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.980905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 534.551348] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 536.033399] LustreError: 19460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.157201] LustreError: 19459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.265289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.774292] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 549.850790] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 553.233337] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 557.635132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 562.755309] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 568.821924] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 586.729547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 586.729714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.736164] LustreError: Skipped 2 previous similar messages [ 586.738854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 586.747803] Lustre: Skipped 1 previous similar message [ 586.766639] Lustre: Skipped 1 previous similar message [ 590.971259] Lustre: server umount lustre-OST0000 complete [ 596.962731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.974043] Lustre: Skipped 1 previous similar message [ 599.639168] Lustre: server umount lustre-MDT0000 complete [ 601.895365] LustreError: 19441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903524 with bad export cookie 8004714741722543535 [ 601.899561] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 601.904672] LustreError: 19441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 602.124324] Lustre: server umount lustre-MDT0001 complete [ 608.774477] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 610.854312] Key type lgssc unregistered [ 611.093667] LNet: 23010:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.099704] LNetError: 23010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 611.116795] LNet: Removed LNI 192.168.203.124@tcp [ 611.852143] Key type .llcrypt unregistered [ 611.854128] Key type ._llcrypt unregistered [ 627.032716] Key type ._llcrypt registered [ 627.035306] Key type .llcrypt registered [ 627.098803] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 627.744639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 627.839393] alg: No test for adler32 (adler32-zlib) [ 628.783552] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 628.940509] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 630.575173] Key type lgssc registered [ 631.366962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 637.163639] loop: module loaded [ 637.232868] loop0: detected capacity change from 0 to 204800000 [ 637.398334] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.407406] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.630937] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 641.571256] loop0: detected capacity change from 0 to 204800000 [ 641.679857] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 641.690928] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 641.700704] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 644.830984] loop0: detected capacity change from 0 to 204800000 [ 648.143511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 648.160231] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.303091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 649.344236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 649.406631] Lustre: lustre-MDT0000: new disk, initializing [ 649.460596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 649.476907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 652.307246] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 658.535595] loop1: detected capacity change from 0 to 204800000 [ 662.460187] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 662.633704] Lustre: lustre-OST07c6: new disk, initializing [ 662.637449] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 662.641057] Lustre: Skipped 1 previous similar message [ 662.644852] Lustre: 26174:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 662.696451] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 666.958281] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 667.711690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 667.721189] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 667.753923] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 671.582464] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 671.593384] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 671.598689] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 671.604969] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 671.610434] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 671.615762] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 687.119423] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 687.123551] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 687.126881] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 687.133452] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 687.138857] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 687.143412] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 691.444166] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 691.459449] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 691.472163] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 691.476455] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.484395] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 691.492092] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.497213] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 691.504172] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.509494] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 691.514595] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.520387] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 691.527095] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.666441] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 694.674477] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 694.681849] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 694.690475] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.700142] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 694.706487] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.711595] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 694.716820] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.720849] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 694.726567] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 694.733398] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 694.741600] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.985977] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 698.995619] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 699.003210] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 699.011993] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.018444] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 699.023658] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.028392] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 699.033151] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.040477] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 699.044682] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.049554] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 699.053787] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.470921] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 708.476762] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 708.481457] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 708.485633] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 708.491084] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 708.498140] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 708.504390] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 708.509694] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 708.520316] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 708.529582] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 708.536335] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 708.543114] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 759.781369] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 759.790814] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 759.806107] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 761.354363] Lustre: server umount lustre-OST07c6 complete [ 767.520163] Lustre: server umount lustre-MDT0000 complete [ 775.737086] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 777.289621] Key type lgssc unregistered [ 777.462656] LNet: 28731:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.466955] LNetError: 28731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 778.537119] LNet: Removed LNI 192.168.203.124@tcp [ 779.066922] Key type .llcrypt unregistered [ 779.068707] Key type ._llcrypt unregistered [ 789.910105] Key type ._llcrypt registered [ 789.913391] Key type .llcrypt registered [ 789.976476] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 790.443995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 790.528594] alg: No test for adler32 (adler32-zlib) [ 791.459427] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 791.617931] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 793.263313] Key type lgssc registered [ 793.833660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 798.611623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 798.625691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 799.883588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 802.918200] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 809.303148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.607353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 812.618688] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 817.301992] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 820.550530] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 824.766198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 824.931911] LustreError: 31799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 824.950192] LustreError: 31799:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 825.040890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.879515] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 830.449582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 833.287618] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 845.792617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 845.799333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 845.817824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 849.893451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 849.897874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 849.899971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 850.966248] Lustre: server umount lustre-OST0000 complete [ 855.009397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.010318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 855.021100] Lustre: Skipped 1 previous similar message [ 855.023485] Lustre: Skipped 1 previous similar message [ 860.138346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 860.142178] Lustre: Skipped 2 previous similar messages [ 865.250976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 865.257116] Lustre: Skipped 1 previous similar message [ 868.319144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 868.495646] Lustre: server umount lustre-MDT0000 complete [ 870.368368] LustreError: 29981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 870.381390] LustreError: 29981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 870.923444] LustreError: 29963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903793 with bad export cookie 428201438209603482 [ 870.927554] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 870.935190] LustreError: 29963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 871.234338] Lustre: server umount lustre-MDT0001 complete [ 878.594632] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 880.682882] Key type lgssc unregistered [ 880.893547] LNet: 33714:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.905563] LNetError: 33714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.940299] LNet: Removed LNI 192.168.203.124@tcp [ 881.569460] Key type .llcrypt unregistered [ 881.572519] Key type ._llcrypt unregistered [ 893.653806] Key type ._llcrypt registered [ 893.660876] Key type .llcrypt registered [ 893.779422] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 894.485195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 894.517227] alg: No test for adler32 (adler32-zlib) [ 895.532945] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 895.734899] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 897.383190] Key type lgssc registered [ 898.189865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 903.484511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 903.504908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 904.787856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 907.549124] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 912.306323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 912.516651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 914.953542] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 918.851321] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 922.281977] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 926.242433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 926.470428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.915695] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 930.599870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 934.308732] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 946.146633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 946.156299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 946.164404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 948.193569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 948.194923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 948.203449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 950.388775] Lustre: server umount lustre-OST0000 complete [ 953.314303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 953.314770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.328316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 953.332467] Lustre: Skipped 2 previous similar messages [ 958.391628] Lustre: server umount lustre-MDT0000 complete [ 958.432357] LustreError: 34960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.441029] LustreError: 34960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 960.185104] LustreError: 34947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903882 with bad export cookie 17096590473033032507 [ 960.187538] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 960.191759] LustreError: 34947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 960.359374] Lustre: server umount lustre-MDT0001 complete [ 965.516992] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 966.960245] Key type lgssc unregistered [ 967.118368] LNet: 38494:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.123593] LNetError: 38494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.135429] LNet: Removed LNI 192.168.203.124@tcp [ 967.607376] Key type .llcrypt unregistered [ 967.613185] Key type ._llcrypt unregistered [ 979.407556] Key type ._llcrypt registered [ 979.409257] Key type .llcrypt registered [ 979.461200] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 980.024485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 980.041067] alg: No test for adler32 (adler32-zlib) [ 980.975360] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 981.108289] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 982.703196] Key type lgssc registered [ 983.340543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 987.348833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 987.361881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 988.630892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.597315] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 995.021555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 995.244767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 997.704952] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1001.646632] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1004.201599] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1007.430143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1007.629475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1010.397534] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1012.776549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1014.355399] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1024.996551] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1024.999344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1025.004195] Lustre: Skipped 2 previous similar messages [ 1025.020497] Lustre: Skipped 2 previous similar messages [ 1027.010972] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 1029.251841] Lustre: server umount lustre-MDT0000 complete [ 1030.113753] LustreError: 39885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1030.128296] LustreError: 39885:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1031.510472] LustreError: 39869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903953 with bad export cookie 11436645758968435612 [ 1031.518652] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1031.520528] LustreError: 39869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1032.108883] LustreError: 39883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1032.119169] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.24@tcp (stopping) [ 1035.233582] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.234130] LustreError: 39885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1035.243650] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1037.714759] Lustre: server umount lustre-MDT0001 complete [ 1056.363210] Lustre: server umount lustre-OST0000 complete [ 1064.216545] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1065.588682] Key type lgssc unregistered [ 1065.756531] LNet: 43746:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.759904] LNetError: 43746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1065.776868] LNet: Removed LNI 192.168.203.124@tcp [ 1066.187629] Key type .llcrypt unregistered [ 1066.189626] Key type ._llcrypt unregistered [ 1077.196543] Key type ._llcrypt registered [ 1077.197969] Key type .llcrypt registered [ 1077.241700] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1077.598827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1077.610762] alg: No test for adler32 (adler32-zlib) [ 1078.509833] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1078.629877] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1080.239153] Key type lgssc registered [ 1080.774916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1084.836517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1084.848814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1086.109813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1088.660290] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1093.662819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.866728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1095.878542] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1099.307147] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1101.933251] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1105.266814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1105.463878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.756110] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1111.592214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1112.574608] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1121.760373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1121.766392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.774577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1124.832749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1124.837570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1124.838776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.399884] Lustre: server umount lustre-OST0000 complete [ 1145.000699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1145.013125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1145.016855] Lustre: Skipped 3 previous similar messages [ 1148.094635] Lustre: server umount lustre-MDT0000 complete [ 1149.915814] LustreError: 45123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904072 with bad export cookie 12224331777148047162 [ 1149.925736] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1164.255216] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1164.440850] Lustre: server umount lustre-MDT0001 complete [ 1170.104757] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1171.700895] Key type lgssc unregistered [ 1171.900252] LNet: 48848:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.904985] LNetError: 48848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1171.923065] LNet: Removed LNI 192.168.203.124@tcp [ 1172.415454] Key type .llcrypt unregistered [ 1172.417426] Key type ._llcrypt unregistered [ 1184.736591] Key type ._llcrypt registered [ 1184.740661] Key type .llcrypt registered [ 1184.797564] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1185.216182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1185.313051] alg: No test for adler32 (adler32-zlib) [ 1186.232483] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1186.384987] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1187.999145] Key type lgssc registered [ 1188.525965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1192.333263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1192.343735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1193.601189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1195.816980] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1200.381915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1200.571951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1202.810352] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1206.878157] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1209.784831] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1213.547694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1213.760416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1216.783226] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1217.835876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1220.376583] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1228.587208] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1230.592540] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1243.076464] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0817h49m22s [ 1245.317125] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1248.184352] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 1249.193096] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 1250.790965] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.798336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1250.809202] Lustre: Skipped 2 previous similar messages [ 1250.826708] Lustre: Skipped 2 previous similar messages [ 1253.940696] Lustre: server umount lustre-MDT0000 complete [ 1254.257435] LustreError: 51672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1254.277815] LustreError: 51672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1257.637958] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1257.965930] LustreError: 51673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.983608] LustreError: 51673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1260.469431] LustreError: 50238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1264.099522] LustreError: 50243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1264.189642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1264.263887] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1264.418301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1264.495454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1268.254154] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1270.764072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1271.522295] Lustre: DEBUG MARKER: Wait for df (36626) ... [ 1272.969306] Lustre: DEBUG MARKER: done [ 1280.997627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1281.000051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1281.010935] Lustre: Skipped 1 previous similar message [ 1281.018139] Lustre: Skipped 3 previous similar messages [ 1286.116357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1286.122483] Lustre: Skipped 2 previous similar messages [ 1294.303515] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1294.405912] Lustre: server umount lustre-OST0000 complete [ 1301.472846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.473948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1301.476498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1301.476504] Lustre: Skipped 2 previous similar messages [ 1301.479454] Lustre: Skipped 1 previous similar message [ 1303.355723] Lustre: server umount lustre-MDT0000 complete [ 1308.100082] LustreError: 50224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904230 with bad export cookie 10126306023745790680 [ 1308.103360] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1308.113834] LustreError: 50224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1308.397928] Lustre: server umount lustre-MDT0001 complete [ 1323.082567] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1326.104415] Key type lgssc unregistered [ 1326.594946] LNet: 55162:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.602041] LNetError: 55162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1327.669160] LNet: Removed LNI 192.168.203.124@tcp [ 1329.421787] Key type .llcrypt unregistered [ 1329.434246] Key type ._llcrypt unregistered [ 1380.105997] Key type ._llcrypt registered [ 1380.110475] Key type .llcrypt registered [ 1380.234832] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1381.508041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1381.543357] alg: No test for adler32 (adler32-zlib) [ 1382.674694] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1382.920864] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1384.591352] Key type lgssc registered [ 1385.825145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1393.107463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1393.137590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.370983] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1394.388582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1394.507489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1397.897915] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1406.707816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1406.761886] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1406.784329] Lustre: 57374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1407.054208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1411.664612] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1419.491287] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1425.847819] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1431.896536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1432.145692] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1432.272962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1438.044532] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1441.881750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1445.269760] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1451.218694] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0817h52m51s [ 1452.830153] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1459.781409] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1462.370469] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1462.414661] LustreError: 57381:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1482.718675] Lustre: lustre-MDT0000: Client e6182d5d-a5e9-4bb8-945d-46fe850e7912 (at 192.168.203.24@tcp) reconnecting [ 1502.423136] LustreError: 57381:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1504.661198] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1507.071861] Lustre: DEBUG MARKER: done [ 1513.443118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1513.456856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1513.476985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1513.954106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1513.960061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1513.987895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1516.533734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1521.634615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1521.642469] Lustre: Skipped 2 previous similar messages [ 1526.755160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1526.761115] Lustre: Skipped 1 previous similar message [ 1527.263122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1527.402750] Lustre: server umount lustre-OST0000 complete [ 1531.887271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.905114] Lustre: Skipped 1 previous similar message [ 1537.015442] Lustre: server umount lustre-MDT0000 complete [ 1540.517349] LustreError: 57367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904462 with bad export cookie 15227286286042378772 [ 1540.524990] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1540.540198] LustreError: 57367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1540.836438] Lustre: server umount lustre-MDT0001 complete [ 1550.010683] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1552.753546] Key type lgssc unregistered [ 1553.097489] LNet: 61703:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.103417] LNetError: 61703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1553.119764] LNet: Removed LNI 192.168.203.124@tcp [ 1554.013378] Key type .llcrypt unregistered [ 1554.015592] Key type ._llcrypt unregistered [ 1598.079796] Key type ._llcrypt registered [ 1598.086371] Key type .llcrypt registered [ 1598.206838] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1599.144800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1599.166601] alg: No test for adler32 (adler32-zlib) [ 1600.343724] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1600.644359] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1602.399173] Key type lgssc registered [ 1603.773339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1607.008479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1607.040161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1608.323146] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1608.377161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1608.625088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1609.602832] Lustre: Failing over lustre-MDT0000 [ 1609.847532] Lustre: server umount lustre-MDT0000 complete [ 1616.722480] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:55:36 (1788904536) [ 1623.645134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1624.161236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.368945] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1641.058806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1641.228824] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1641.291442] Lustre: 64507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1641.705770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1647.074660] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1655.422740] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1662.079892] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1669.257405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1669.531527] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1669.761459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1676.065504] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1676.949225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1684.250398] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1690.997804] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1702.908500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.910920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1702.932298] Lustre: Skipped 2 previous similar messages [ 1702.956569] Lustre: Skipped 2 previous similar messages [ 1705.978770] Lustre: server umount lustre-MDT0000 complete [ 1709.832680] LustreError: 64501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904632 with bad export cookie 14548962122096193830 [ 1709.837423] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1709.851340] LustreError: 64501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1710.183416] Lustre: server umount lustre-MDT0001 complete [ 1713.873800] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1718.505565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1729.512910] Lustre: 63435:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904635/real 1788904635] req@00000000034110bc x1875802348102912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1729.618787] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1730.527114] Lustre: 63433:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904636/real 1788904636] req@00000000c8750a89 x1875802348103040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904652 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1730.815852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.687491] Lustre: 63433:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904640/real 1788904640] req@000000009a995870 x1875802348103168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904656 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1734.818227] LustreError: 63431:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004d0ee07e x1875802348103936/t0(0) o250->MGC192.168.203.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1735.354050] LustreError: 66359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1735.404051] LustreError: 66359:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1735.517592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1735.603210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1737.580227] LustreError: 66360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1740.249256] LustreError: 68278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1740.302262] LustreError: 68278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1741.256215] Lustre: 63435:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904647/real 1788904647] req@00000000b911a8c9 x1875802348103552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904663 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1741.327077] Lustre: 63435:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1741.548936] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1745.378432] LustreError: 68279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1745.409713] LustreError: 68279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1750.499246] LustreError: 68278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1750.564952] LustreError: 68278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1754.018447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1754.697587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1757.803493] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1759.248258] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1767.965031] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1776.008686] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1785.324162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1785.340221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1785.371578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1788.901591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1788.907199] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1788.933668] Lustre: Skipped 1 previous similar message [ 1791.766806] Lustre: server umount lustre-MDT0000 complete [ 1794.019626] LustreError: 69578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1794.041839] LustreError: 69578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1796.208909] LustreError: 65244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904718 with bad export cookie 14548962122096196105 [ 1796.217651] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1796.227356] LustreError: 65244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1796.663953] Lustre: server umount lustre-MDT0001 complete [ 1803.931220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1814.095591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.519156] Lustre: 63435:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904721/real 1788904721] req@000000000f89fdb3 x1875802348133888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904737 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1815.547593] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.950820] LustreError: 69015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1822.033479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1822.095478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1825.547620] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1827.104583] Lustre: 63435:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788904733/real 1788904733] req@00000000b6ce4062 x1875802348134528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788904749 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1827.134640] Lustre: 63435:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1832.997710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1837.227634] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1838.565593] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1838.572386] Lustre: Skipped 1 previous similar message [ 1843.499279] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1844.780233] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1848.283419] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1853.921536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1853.933218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.945257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1853.950281] Lustre: Skipped 2 previous similar messages [ 1856.089468] Lustre: server umount lustre-MDT0000 complete [ 1859.158418] LustreError: 64502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904781 with bad export cookie 14548962122096197484 [ 1859.159263] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1859.183091] LustreError: 64502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1859.426192] Lustre: server umount lustre-MDT0001 complete [ 1864.600344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1867.831466] Lustre: DEBUG MARKER: files compared the same [ 1879.196763] Lustre: server umount lustre-OST0000 complete [ 1891.011941] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1893.230140] Key type lgssc unregistered [ 1893.542982] LNet: 74238:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.549504] LNetError: 74238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.565923] LNet: Removed LNI 192.168.203.124@tcp [ 1894.305895] Key type .llcrypt unregistered [ 1894.310898] Key type ._llcrypt unregistered [ 1913.041861] Key type ._llcrypt registered [ 1913.043997] Key type .llcrypt registered [ 1913.124828] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1913.882337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1913.898956] alg: No test for adler32 (adler32-zlib) [ 1914.847949] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1915.030305] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1916.680103] Key type lgssc registered [ 1917.539892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1923.745359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1923.768983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.148535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1929.087836] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug +malloc all [ 1935.332232] LustreError: 75636:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.351373] LustreError: 75636:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1936.249502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1936.522322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1940.599469] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug +malloc all [ 1946.572558] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1951.087242] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1956.405352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1956.631331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.167486] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug +malloc all [ 1966.262158] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1969.970200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1980.385208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1980.389729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.390646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1980.409304] Lustre: Skipped 1 previous similar message [ 1984.013295] Lustre: server umount lustre-OST0000 complete [ 1988.064755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1988.070793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.082294] Lustre: Skipped 1 previous similar message [ 1988.086445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1990.624086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1990.626881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1992.826799] Lustre: server umount lustre-MDT0000 complete [ 1995.421224] LustreError: 75617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904917 with bad export cookie 14539725325891318696 [ 1995.427918] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1995.432325] LustreError: 75617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1995.601091] Lustre: server umount lustre-MDT0001 complete [ 2003.713810] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2005.769671] Key type lgssc unregistered [ 2006.022562] LNet: 79200:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.028358] LNetError: 79200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2006.038463] LNet: Removed LNI 192.168.203.124@tcp [ 2006.674854] Key type .llcrypt unregistered [ 2006.677939] Key type ._llcrypt unregistered [ 2022.706964] Key type ._llcrypt registered [ 2022.709181] Key type .llcrypt registered [ 2022.808311] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2023.510872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2023.570249] alg: No test for adler32 (adler32-zlib) [ 2024.540427] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2024.745209] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2026.384252] Key type lgssc registered [ 2027.263759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2033.070557] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2033.086649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2048.607363] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2053.728364] LustreError: 80570:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.124@tcp: failed processing log, type 4: rc = -110 [ 2080.351269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2084.185063] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2089.259656] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2094.336585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2094.419654] Lustre: *** cfs_fail_loc=706, val=0*** [ 2094.721215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2094.784440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2097.601892] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2101.231573] LustreError: 81656:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2101.250937] LustreError: 81656:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2102.791595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2103.042749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2105.785325] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2110.413110] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2113.862764] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2115.555835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2115.560539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.571377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2118.626486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2118.627536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2118.629706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2118.634615] Lustre: Skipped 1 previous similar message [ 2121.325447] Lustre: server umount lustre-OST0000 complete [ 2123.745891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.748522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.751194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2123.756687] Lustre: Skipped 1 previous similar message [ 2123.768434] Lustre: Skipped 1 previous similar message [ 2128.869543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2128.876142] Lustre: Skipped 1 previous similar message [ 2129.823263] Lustre: server umount lustre-MDT0000 complete [ 2131.900560] LustreError: 80577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905054 with bad export cookie 9753802194909307914 [ 2131.905907] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2131.908832] LustreError: 80577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2132.239604] Lustre: server umount lustre-MDT0001 complete [ 2138.576486] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2140.481445] Key type lgssc unregistered [ 2140.726633] LNet: 84182:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2140.732815] LNetError: 84182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2140.748237] LNet: Removed LNI 192.168.203.124@tcp [ 2141.327524] Key type .llcrypt unregistered [ 2141.332344] Key type ._llcrypt unregistered [ 2156.882633] Key type ._llcrypt registered [ 2156.884845] Key type .llcrypt registered [ 2156.964151] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2157.570733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2157.621694] alg: No test for adler32 (adler32-zlib) [ 2158.550905] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2158.703991] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2160.335160] Key type lgssc registered [ 2160.943658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2165.350643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2165.369274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2169.134643] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2173.841317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2174.172229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2176.624944] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2179.554341] LustreError: 86265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.566738] LustreError: 86265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2181.710857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2181.971789] LustreError: 86264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2181.983545] LustreError: 86264:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2181.986611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2184.032755] LustreError: 86269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2184.053061] LustreError: 86269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2185.395791] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2189.283498] LustreError: 86268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.293754] LustreError: 86268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2190.905988] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2190.914961] Lustre: MGS: Client 2658664e-a244-47fd-9ea6-a3efe51e05e6 (at 0@lo) reconnecting [ 2190.934355] LustreError: 86279:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2191.068402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2191.112221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2193.624253] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2206.691668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2206.694432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.702432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2206.706568] Lustre: Skipped 1 previous similar message [ 2209.906314] Lustre: server umount lustre-OST0000 complete [ 2216.929144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.933197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2216.937496] Lustre: Skipped 1 previous similar message [ 2216.937771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2216.950095] Lustre: Skipped 2 previous similar messages [ 2218.072747] Lustre: server umount lustre-MDT0000 complete [ 2220.243809] Lustre: server umount lustre-MDT0001 complete [ 2222.221730] Lustre: server umount MGS complete [ 2228.252709] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:05:49 (1788905149) [ 2242.794753] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 2247.352378] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2262.770876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.593726] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2269.170424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2269.269487] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2269.292738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2269.335671] Lustre: lustre-MDT0001: new disk, initializing [ 2269.369889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2269.380426] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2269.383812] Lustre: 93423:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2269.389483] LustreError: 93423:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2270.984650] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2276.101163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2276.230223] Lustre: lustre-OST0000: new disk, initializing [ 2276.232755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2276.237205] Lustre: 94189:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2276.272363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2278.369380] Lustre: 94319:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000017d15785 x1875802933203200/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788905216 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2279.097288] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2283.722676] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2286.818876] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2286.826676] Lustre: MGS: Client 1969e4de-f9c0-42c6-bd74-404709ffdce2 (at 0@lo) reconnecting [ 2286.844379] LustreError: 93408:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2286.855601] Lustre: 92693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2286.875963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2286.917973] Lustre: lustre-MDT0000: new disk, initializing [ 2287.002584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2287.014967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2287.456507] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2288.032349] Lustre: 95258:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000061a7ea8f x1875802933209344/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788905226 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2288.803562] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2297.123859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2297.131912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2297.201440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2319.327866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2319.331063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2319.337212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2324.695523] Lustre: server umount lustre-OST0000 complete [ 2331.104803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2331.104884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.105853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2331.105857] Lustre: Skipped 2 previous similar messages [ 2331.116572] Lustre: Skipped 2 previous similar messages [ 2332.362514] Lustre: server umount lustre-MDT0000 complete [ 2340.074428] Lustre: server umount lustre-MDT0001 complete [ 2341.388472] Lustre: server umount MGS complete [ 2345.639626] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:07:46 (1788905266) [ 2351.667929] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2352.700089] Key type lgssc unregistered [ 2352.839308] LNet: 97751:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2352.841791] LNetError: 97751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2352.848251] LNet: Removed LNI 192.168.203.124@tcp [ 2353.159595] Key type .llcrypt unregistered [ 2353.161457] Key type ._llcrypt unregistered [ 2360.272806] Key type ._llcrypt registered [ 2360.274816] Key type .llcrypt registered [ 2360.311504] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2360.667575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2360.719822] alg: No test for adler32 (adler32-zlib) [ 2361.590953] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2361.691943] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2363.280145] Key type lgssc registered [ 2363.716292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2366.710695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2366.718559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.899223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.389322] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2372.492131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2372.619329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2374.143925] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2376.560818] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2378.524246] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2380.805560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2380.932737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2382.859237] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2385.432260] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2393.253432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2398.688726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.689136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2398.690024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2398.692880] Lustre: Skipped 1 previous similar message [ 2403.809501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2403.812340] Lustre: Skipped 2 previous similar messages [ 2404.626225] Lustre: server umount lustre-OST0000 complete [ 2408.415765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2408.418489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2408.422368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2414.047922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2414.049636] Lustre: Skipped 2 previous similar messages [ 2419.168394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2419.170534] Lustre: Skipped 1 previous similar message [ 2420.191117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2420.256252] Lustre: server umount lustre-MDT0000 complete [ 2421.508117] LustreError: 98931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905343 with bad export cookie 10437762064346995189 [ 2421.510837] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2421.511474] LustreError: 98931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2421.630785] Lustre: server umount lustre-MDT0001 complete [ 2425.157628] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2426.115080] Key type lgssc unregistered [ 2426.240562] LNet: 102502:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2426.243759] LNetError: 102502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2426.255299] LNet: Removed LNI 192.168.203.124@tcp [ 2426.562894] Key type .llcrypt unregistered [ 2426.563984] Key type ._llcrypt unregistered [ 2432.845772] Key type ._llcrypt registered [ 2432.847209] Key type .llcrypt registered [ 2432.881877] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2433.211411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2433.246259] alg: No test for adler32 (adler32-zlib) [ 2434.084471] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2434.165603] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2435.735154] Key type lgssc registered [ 2436.043510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2437.831426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2437.831486] LustreError: 103564:0:(tgt_mount.c:2659:server_fill_super()) cfs_race id 716 sleeping [ 2437.833972] Lustre: Skipped 1 previous similar message [ 2437.838316] LustreError: 103563:0:(tgt_mount.c:2659:server_fill_super()) cfs_fail_race id 716 waking [ 2437.840535] LustreError: 103564:0:(tgt_mount.c:2659:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 2437.843112] LustreError: 103564:0:(tgt_mount.c:2671:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2437.845804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2437.847095] LustreError: 103564:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2438.988388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2441.375406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2441.456842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2442.663330] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2444.221317] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing lsmod [ 2445.163052] LustreError: 104771:0:(tgt_mount.c:2659:server_fill_super()) cfs_race id 716 sleeping [ 2445.163096] LustreError: 104770:0:(tgt_mount.c:2659:server_fill_super()) cfs_fail_race id 716 waking [ 2445.168273] LustreError: 104771:0:(tgt_mount.c:2659:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 2445.170641] LustreError: 104771:0:(tgt_mount.c:2671:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2445.173797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2445.175818] LustreError: 104771:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2445.288827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2446.815927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2446.818601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2446.823298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.400318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.401201] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.403394] Lustre: Skipped 1 previous similar message [ 2450.407375] Lustre: Skipped 1 previous similar message [ 2451.424791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2451.426818] Lustre: Skipped 3 previous similar messages [ 2452.365533] Lustre: server umount lustre-MDT0000 complete [ 2453.492204] LustreError: 103575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905375 with bad export cookie 8772923587114882922 [ 2453.495176] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2453.496233] LustreError: 103575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2455.519252] LustreError: 104031:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000dcb4be2c x1875803222588416/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2455.566620] Lustre: server umount lustre-MDT0001 complete [ 2465.940291] Lustre: server umount lustre-OST0000 complete [ 2468.773192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2468.918350] LustreError: 105830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.924997] LustreError: 105830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2468.943997] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2470.298751] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2472.930471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2473.044284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2474.385641] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2476.726911] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2478.496176] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2480.647845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2482.602455] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2485.012214] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2494.054326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2498.528689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2498.530908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.534527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2499.554501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2502.797886] Lustre: server umount lustre-OST0000 complete [ 2504.672673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.677174] Lustre: Skipped 1 previous similar message [ 2504.679571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2504.681440] Lustre: Skipped 3 previous similar messages [ 2510.161617] Lustre: server umount lustre-MDT0000 complete [ 2511.424379] LustreError: 105811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905433 with bad export cookie 8772923587114883580 [ 2511.426548] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2511.429778] LustreError: 105811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2511.551866] Lustre: server umount lustre-MDT0001 complete [ 2515.100063] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2516.074572] Key type lgssc unregistered [ 2516.187434] LNet: 109379:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2516.190936] LNetError: 109379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2516.198365] LNet: Removed LNI 192.168.203.124@tcp [ 2516.487462] Key type .llcrypt unregistered [ 2516.488556] Key type ._llcrypt unregistered [ 2525.024641] Key type ._llcrypt registered [ 2525.025791] Key type .llcrypt registered [ 2525.058896] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2525.459826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2525.466886] alg: No test for adler32 (adler32-zlib) [ 2526.313114] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2526.390990] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2527.967148] Key type lgssc registered [ 2528.312378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2530.737526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2530.743356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.874355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2533.133254] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2535.867344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2535.994735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2537.279636] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2539.567530] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2541.280240] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2543.430274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2543.542310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2545.455605] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2547.822861] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2556.838064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2560.246883] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2560.988496] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2562.017799] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2562.017910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2562.026374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2562.028294] Lustre: Skipped 1 previous similar message [ 2567.141105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2567.143112] Lustre: Skipped 1 previous similar message [ 2572.256511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2572.259132] Lustre: Skipped 1 previous similar message [ 2576.351106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2576.400344] Lustre: server umount lustre-OST0000 complete [ 2581.983845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2581.986085] LustreError: Skipped 1 previous similar message [ 2581.987634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2581.990904] Lustre: Skipped 1 previous similar message [ 2581.992516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2583.758676] Lustre: server umount lustre-MDT0000 complete [ 2584.968229] LustreError: 110800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905507 with bad export cookie 6060998010315023199 [ 2584.971072] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2584.971516] LustreError: 110800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2585.082460] Lustre: server umount lustre-MDT0001 complete [ 2588.517204] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2589.555638] Key type lgssc unregistered [ 2589.688453] LNet: 114428:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2589.691548] LNetError: 114428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2589.698355] LNet: Removed LNI 192.168.203.124@tcp [ 2590.001295] Key type .llcrypt unregistered [ 2590.003220] Key type ._llcrypt unregistered [ 2596.576812] Key type ._llcrypt registered [ 2596.578167] Key type .llcrypt registered [ 2596.613964] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2596.911317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2596.927690] alg: No test for adler32 (adler32-zlib) [ 2597.779294] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2597.868287] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2599.455247] Key type lgssc registered [ 2599.830673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2602.504872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2602.510439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2603.657280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.099851] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2607.951878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2608.077292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2609.394245] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2611.919536] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2613.724908] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2615.988675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2616.103422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2617.977262] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2620.362437] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2628.389936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2633.698770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2633.698884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2633.700130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2633.701954] LustreError: Skipped 1 previous similar message [ 2633.707225] Lustre: Skipped 1 previous similar message [ 2638.818889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2638.821075] Lustre: Skipped 2 previous similar messages [ 2639.763808] Lustre: server umount lustre-OST0000 complete [ 2643.936510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2643.936955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2643.940804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2643.945895] Lustre: Skipped 1 previous similar message [ 2643.951652] Lustre: Skipped 1 previous similar message [ 2647.428535] Lustre: server umount lustre-MDT0000 complete [ 2649.060156] LustreError: 115626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2649.073175] LustreError: 115626:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2649.734273] LustreError: 115607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905571 with bad export cookie 2774214627739115961 [ 2649.735488] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2649.743026] LustreError: 115607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2650.594638] Lustre: server umount lustre-MDT0001 complete [ 2657.384278] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2658.945592] Key type lgssc unregistered [ 2659.097434] LNet: 119178:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2659.103370] LNetError: 119178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2659.114502] LNet: Removed LNI 192.168.203.124@tcp [ 2659.658516] Key type .llcrypt unregistered [ 2659.659972] Key type ._llcrypt unregistered [ 2678.776903] Key type ._llcrypt registered [ 2678.780409] Key type .llcrypt registered [ 2678.926245] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2680.127192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2680.151476] alg: No test for adler32 (adler32-zlib) [ 2681.165404] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2681.421565] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2683.071164] Key type lgssc registered [ 2684.453768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2692.059132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2692.084711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2693.655971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2697.669415] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2702.824780] LustreError: 120572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.846675] LustreError: 120572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2703.657988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2703.877646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2706.586267] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2710.407792] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2713.147140] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2716.711234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2716.873616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2717.928090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2719.315751] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2722.788892] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2730.629299] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2731.974099] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 2732.250703] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2732.253896] Lustre: Skipped 2 previous similar messages [ 2734.401465] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2734.405284] Lustre: Skipped 2 previous similar messages [ 2739.211971] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2739.234514] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2739.238018] Lustre: Skipped 1 previous similar message [ 2743.274269] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.24@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2743.285779] Lustre: Skipped 2 previous similar messages [ 2746.197425] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.24@tcp 192.168.0.[2,10]@tcp [ 2748.462643] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.24/24@tcp in log lustre-MDT0000 [ 2748.467988] Lustre: Skipped 2 previous similar messages [ 2753.546726] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.24/24@tcp [ 2753.551772] Lustre: Skipped 1 previous similar message [ 2759.136466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2759.141282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2759.149930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2759.670429] Lustre: server umount lustre-OST0000 complete [ 2765.282290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2765.284988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.288309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2765.291433] Lustre: Skipped 1 previous similar message [ 2765.303096] Lustre: Skipped 1 previous similar message [ 2767.651880] Lustre: server umount lustre-MDT0000 complete [ 2769.403210] LustreError: 123409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905691 with bad export cookie 17688834033134014135 [ 2769.407730] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2769.413775] LustreError: 123409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2779.682944] Lustre: server umount lustre-MDT0001 complete [ 2784.130745] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2785.410327] Key type lgssc unregistered [ 2785.564747] LNet: 125609:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2785.569750] LNetError: 125609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2785.585395] LNet: Removed LNI 192.168.203.124@tcp [ 2786.021305] Key type .llcrypt unregistered [ 2786.023230] Key type ._llcrypt unregistered [ 2799.915558] Key type ._llcrypt registered [ 2799.917349] Key type .llcrypt registered [ 2799.980093] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2800.575548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2800.593554] alg: No test for adler32 (adler32-zlib) [ 2801.528160] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2801.677586] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2803.279185] Key type lgssc registered [ 2803.855447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2807.778478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2807.789106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2809.025137] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2809.032854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2809.037994] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.24/24@tcp [ 2811.050772] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2815.683117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.867776] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2815.881892] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.24/24@tcp [ 2815.910251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2818.105716] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2821.953665] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2824.883102] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2828.487748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2828.701789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2831.950643] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2833.834996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2835.691218] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2846.178775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2846.185264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.190953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2846.193149] Lustre: Skipped 1 previous similar message [ 2850.349860] Lustre: server umount lustre-OST0000 complete [ 2856.418653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2856.425738] Lustre: Skipped 1 previous similar message [ 2856.429735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2856.432402] Lustre: Skipped 1 previous similar message [ 2858.412796] Lustre: server umount lustre-MDT0000 complete [ 2860.160474] LustreError: 127127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905782 with bad export cookie 11953377492114405292 [ 2860.162281] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2860.171902] LustreError: 127127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2860.366673] Lustre: server umount lustre-MDT0001 complete [ 2865.468359] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2867.168563] Key type lgssc unregistered [ 2867.356620] LNet: 130730:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2867.366323] LNetError: 130730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2867.382719] LNet: Removed LNI 192.168.203.124@tcp [ 2867.962468] Key type .llcrypt unregistered [ 2867.966922] Key type ._llcrypt unregistered [ 2883.586856] Key type ._llcrypt registered [ 2883.590830] Key type .llcrypt registered [ 2883.669366] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 2892.101421] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2892.754653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2892.781454] alg: No test for adler32 (adler32-zlib) [ 2893.711233] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2893.861603] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2895.495217] Key type lgssc registered [ 2896.231365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2918.285714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2918.322090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2919.512596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2919.550881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2919.628168] Lustre: lustre-MDT0000: new disk, initializing [ 2919.683660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2919.703522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2922.404069] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2928.978035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.072652] Lustre: 134104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2929.118561] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2929.126018] Lustre: Skipped 1 previous similar message [ 2929.200560] Lustre: lustre-MDT0001: new disk, initializing [ 2929.260502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2929.287373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2929.296443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2931.715802] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2937.406478] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2940.631737] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2944.289329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2944.451344] Lustre: lustre-OST0000: new disk, initializing [ 2944.454186] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2944.460366] Lustre: 136070:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2944.543378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2948.134960] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2952.807621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2952.830946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2952.914722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2957.632731] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2962.486141] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2962.650530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2966.912409] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2967.080515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2973.670935] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2973.671659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.688300] LustreError: Skipped 1 previous similar message [ 2973.693343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2973.712188] Lustre: Skipped 1 previous similar message [ 2978.791917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2978.794894] Lustre: Skipped 1 previous similar message [ 2982.879079] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2983.053054] Lustre: server umount lustre-OST0000 complete [ 2989.025648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2989.065303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2989.067923] Lustre: Skipped 1 previous similar message [ 2994.148891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2994.156479] Lustre: Skipped 1 previous similar message [ 2999.265829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2999.271912] Lustre: Skipped 1 previous similar message [ 3001.311134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3001.410419] Lustre: server umount lustre-MDT0000 complete [ 3003.848298] LustreError: 134098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905926 with bad export cookie 9573038054018380351 [ 3003.852315] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3003.862206] LustreError: 134098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3004.082291] Lustre: server umount lustre-MDT0001 complete [ 3009.651290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3009.963340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3013.028735] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3015.136242] LustreError: 138289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3015.150313] LustreError: 138289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3020.264109] LustreError: 138290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3020.505073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3020.929533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3024.752165] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3030.642744] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3034.683522] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3039.662603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3039.930033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3044.346914] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3049.612667] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3053.242260] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3055.583191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3059.104985] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3059.251688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3065.660432] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3065.805439] Lustre: lustre-OST0001: new disk, initializing [ 3065.807805] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3065.813884] Lustre: 141670:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3065.890595] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3070.724764] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3074.116455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3074.130784] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3074.179875] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3077.222860] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3077.396874] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3080.592136] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3080.792280] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3084.757809] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e6e88b62000.ost_server_uuid 50 [ 3085.753160] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e6e88b62000.ost_server_uuid in FULL state after 0 sec [ 3095.009587] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3095.010316] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3095.018592] LustreError: Skipped 1 previous similar message [ 3095.026810] Lustre: Skipped 1 previous similar message [ 3095.029885] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3095.036459] Lustre: Skipped 1 previous similar message [ 3096.997637] Lustre: server umount lustre-OST0001 complete [ 3100.136284] LustreError: 141675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.136397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.149615] LustreError: 141675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3100.165472] Lustre: Skipped 1 previous similar message [ 3105.257374] LustreError: 141675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3105.274768] LustreError: 141675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3105.759627] Lustre: server umount lustre-OST0000 complete [ 3110.374125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3110.385590] Lustre: Skipped 1 previous similar message [ 3114.634175] LustreError: 143464:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000a2e4f07a) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3114.707730] Lustre: server umount lustre-MDT0000 complete [ 3115.489865] LustreError: 142914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3115.507886] LustreError: 142914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3117.362425] LustreError: 141669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906039 with bad export cookie 9573038054018381233 [ 3117.368288] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3117.372397] LustreError: 141669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3117.637793] Lustre: server umount lustre-MDT0001 complete [ 3129.913482] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 3131.829671] Key type lgssc unregistered [ 3132.062652] LNet: 144673:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3132.068458] LNetError: 144673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3132.087585] LNet: Removed LNI 192.168.203.124@tcp [ 3132.701516] Key type .llcrypt unregistered [ 3132.703963] Key type ._llcrypt unregistered [ 3158.969299] Key type ._llcrypt registered [ 3158.973663] Key type .llcrypt registered [ 3159.071585] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 3168.844157] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3169.536203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3169.548059] alg: No test for adler32 (adler32-zlib) [ 3170.492700] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 3170.658837] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 3172.287566] Key type lgssc registered [ 3173.113895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3193.233145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3193.247564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3194.400173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3194.422525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3194.502642] Lustre: lustre-MDT0000: new disk, initializing [ 3194.560680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3194.580766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3197.548199] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3204.918683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3204.992979] Lustre: 148803:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3205.030459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3205.037429] Lustre: Skipped 1 previous similar message [ 3205.101847] Lustre: lustre-MDT0001: new disk, initializing [ 3205.169252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3205.202335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3205.209436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3208.012987] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3214.386576] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3217.940748] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3221.630287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3221.783566] Lustre: lustre-OST0000: new disk, initializing [ 3221.786719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3221.794506] Lustre: 150770:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3221.860687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.206278] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3230.213544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3230.218994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3230.276681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3231.082993] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3235.247642] Lustre: Failing over lustre-OST0000 [ 3235.309907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3235.310258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3235.327135] Lustre: Skipped 1 previous similar message [ 3235.384757] Lustre: server umount lustre-OST0000 complete [ 3250.888205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3251.085625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3251.106577] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3254.211905] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3254.991736] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3255.235970] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3255.237132] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3257.560069] Lustre: Failing over lustre-MDT0000 [ 3257.698668] Lustre: server umount lustre-MDT0000 complete [ 3259.340963] LustreError: 148810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3259.351433] LustreError: 148810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3260.387271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3260.388503] LustreError: 148814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3260.393942] Lustre: Skipped 1 previous similar message [ 3260.411826] LustreError: 148814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3264.425772] LustreError: 150677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3269.546074] LustreError: 148810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3269.564032] LustreError: 148810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3273.318750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3273.414547] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3273.595737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3273.632906] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3274.194896] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3276.159545] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3278.833404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3278.837633] Lustre: Skipped 1 previous similar message [ 3278.861803] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3278.886480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3283.943328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.950863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3283.954966] Lustre: Skipped 2 previous similar messages [ 3283.962056] Lustre: Skipped 1 previous similar message [ 3287.475545] Lustre: server umount lustre-OST0000 complete [ 3294.177141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3294.178915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3294.189787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3294.199170] LustreError: Skipped 1 previous similar message [ 3294.205577] Lustre: Skipped 1 previous similar message [ 3295.851219] Lustre: server umount lustre-MDT0000 complete [ 3297.856296] LustreError: 148795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906220 with bad export cookie 13782781347436343550 [ 3297.860632] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3297.862955] LustreError: 148795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3298.039906] Lustre: server umount lustre-MDT0001 complete [ 3303.949041] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 3305.625535] Key type lgssc unregistered [ 3305.796744] LNet: 154425:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3305.803190] LNetError: 154425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3305.823308] LNet: Removed LNI 192.168.203.124@tcp [ 3306.313539] Key type .llcrypt unregistered [ 3306.319266] Key type ._llcrypt unregistered [ 3332.313634] Key type ._llcrypt registered [ 3332.315344] Key type .llcrypt registered [ 3332.374087] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3333.015368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3333.074302] alg: No test for adler32 (adler32-zlib) [ 3334.016370] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 3334.198247] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 3335.823222] Key type lgssc registered [ 3336.592033] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3341.532061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3341.545727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.809307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3345.381355] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3351.507484] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3353.059995] LustreError: 155944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3353.072314] LustreError: 155944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3354.116957] LustreError: 155972:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3354.123088] LustreError: 155972:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3354.129932] LustreError: 155972:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 3354.327816] Lustre: server umount lustre-MDT0000 complete [ 3363.264346] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 3368.555608] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3388.666345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3388.802514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3388.822556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3388.866516] Lustre: lustre-MDT0000: new disk, initializing [ 3388.928271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.939114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3391.419283] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3397.498147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3397.579480] Lustre: 159280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3397.618866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3397.623027] Lustre: Skipped 1 previous similar message [ 3397.681862] Lustre: lustre-MDT0001: new disk, initializing [ 3397.745336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3397.767635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3397.775792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3399.957339] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3405.454855] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3408.579353] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3412.867452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3413.050263] Lustre: lustre-OST0000: new disk, initializing [ 3413.054678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3413.060514] Lustre: 161247:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3413.134358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3413.968937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3413.975436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3414.045938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3416.572928] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3422.394372] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3425.976221] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3426.985250] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:25:47 (1788906347) [ 3674.082210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3674.086282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3674.098326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3675.115974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3675.116958] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3675.120639] Lustre: Skipped 1 previous similar message [ 3675.152245] Lustre: Skipped 1 previous similar message [ 3678.663419] Lustre: server umount lustre-MDT0000 complete [ 3680.224263] LustreError: 159290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3680.245269] LustreError: 159290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3682.185499] LustreError: 159270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906604 with bad export cookie 950201078668558075 [ 3682.197264] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3682.201426] LustreError: 159270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3685.345078] LustreError: 159287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3685.345585] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3685.395457] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3685.409115] Lustre: Skipped 1 previous similar message [ 3687.391902] LustreError: 161255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3688.663881] Lustre: server umount lustre-MDT0001 complete [ 3698.686632] Lustre: server umount lustre-OST0000 complete [ 3707.427684] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 3716.778392] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3750.870023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3751.243996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3751.288650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3751.383590] Lustre: lustre-MDT0000: new disk, initializing [ 3751.494341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3751.512742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3756.541828] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3766.916835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3767.067969] Lustre: 165913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3767.120602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3767.127423] Lustre: Skipped 1 previous similar message [ 3767.226403] Lustre: lustre-MDT0001: new disk, initializing [ 3767.342868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3767.405039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3767.420722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3771.603884] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3781.279877] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3787.109124] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3793.573620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3793.914528] Lustre: lustre-OST0000: new disk, initializing [ 3793.921734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3793.930041] Lustre: 167883:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3794.029843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.243982] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3801.728856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3801.746405] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3801.851680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3811.421966] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3816.502387] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3816.679493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3820.776323] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3821.005087] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3827.684863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3827.696702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3827.698281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3827.698288] Lustre: Skipped 1 previous similar message [ 3827.699581] Lustre: Skipped 1 previous similar message [ 3827.724779] LustreError: Skipped 1 previous similar message [ 3828.981296] Lustre: server umount lustre-OST0000 complete [ 3832.801955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.803821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3832.823281] Lustre: Skipped 1 previous similar message [ 3832.829771] Lustre: Skipped 1 previous similar message [ 3838.627915] Lustre: server umount lustre-MDT0000 complete [ 3841.675375] LustreError: 165905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906763 with bad export cookie 950201078668716912 [ 3841.686107] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3841.906323] Lustre: server umount lustre-MDT0001 complete [ 3849.633432] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:32:49 (1788906769) [ 3864.057394] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 3871.608627] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3898.271884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3898.425856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3898.450273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3898.513422] Lustre: lustre-MDT0000: new disk, initializing [ 3898.569582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3898.589853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3901.887891] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3910.902546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3911.028808] Lustre: 172636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3911.385477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3915.047396] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3922.390491] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3926.210564] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3930.205066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3930.433086] Lustre: 174610:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3933.911385] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3940.080375] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3940.376354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3940.382398] Lustre: Skipped 1 previous similar message [ 3940.386386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3940.444412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3945.443623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3945.444116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.449184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3945.449191] Lustre: Skipped 3 previous similar messages [ 3945.461990] Lustre: Skipped 1 previous similar message [ 3949.820628] Lustre: server umount lustre-OST0000 complete [ 3952.609254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3952.621540] LustreError: Skipped 1 previous similar message [ 3958.391369] Lustre: server umount lustre-MDT0000 complete [ 3960.397430] LustreError: 172626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906882 with bad export cookie 950201078668717794 [ 3960.400921] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3960.403600] LustreError: 172626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3960.607825] Lustre: server umount lustre-MDT0001 complete [ 3964.853291] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:34:45 (1788906885) [ 3973.114643] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 3978.114445] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3994.659049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3994.780598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3994.794903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3994.797480] Lustre: Skipped 3 previous similar messages [ 3994.838149] Lustre: lustre-MDT0000: new disk, initializing [ 3994.840296] Lustre: Skipped 2 previous similar messages [ 3994.880079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3994.884795] Lustre: Skipped 2 previous similar messages [ 3996.825691] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4002.481084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4002.539040] Lustre: 178925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4002.579387] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4002.584248] Lustre: Skipped 4 previous similar messages [ 4002.683990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4004.545805] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4009.799179] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4012.507339] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4016.055366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4016.203564] Lustre: 180904:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4017.423606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4017.430188] Lustre: Skipped 2 previous similar messages [ 4017.485982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4019.171880] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4024.868294] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4032.996295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4032.996681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4033.002526] LustreError: Skipped 1 previous similar message [ 4033.010661] Lustre: Skipped 2 previous similar messages [ 4034.409383] Lustre: server umount lustre-OST0000 complete [ 4038.113624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4038.116684] Lustre: Skipped 3 previous similar messages [ 4042.686643] Lustre: server umount lustre-MDT0000 complete [ 4043.232224] LustreError: 179736:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4043.241686] LustreError: 179736:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4044.678735] LustreError: 178917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906966 with bad export cookie 950201078668718879 [ 4044.686646] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4044.689871] LustreError: 178917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4044.844782] Lustre: server umount lustre-MDT0001 complete [ 4051.154130] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 4052.614898] Key type lgssc unregistered [ 4052.788419] LNet: 182758:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4052.796788] LNetError: 182758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4052.813994] LNet: Removed LNI 192.168.203.124@tcp [ 4053.282592] Key type .llcrypt unregistered [ 4053.285134] Key type ._llcrypt unregistered [ 4066.615454] Key type ._llcrypt registered [ 4066.617105] Key type .llcrypt registered [ 4066.673665] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4067.162648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4067.192661] alg: No test for adler32 (adler32-zlib) [ 4068.106189] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4068.231545] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 4069.831182] Key type lgssc registered [ 4070.368061] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4074.245389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4074.256096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4075.483990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4077.208664] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4080.788758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4080.972205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4082.767579] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4086.418419] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4089.205994] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4092.341448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4092.506526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4095.209546] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4098.612165] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4100.717710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4121.570578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4121.571731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4121.572774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4121.576200] Lustre: Skipped 1 previous similar message [ 4124.704741] Lustre: server umount lustre-OST0000 complete [ 4126.689158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4126.689720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4126.695464] Lustre: Skipped 1 previous similar message [ 4126.697794] Lustre: Skipped 1 previous similar message [ 4131.810264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4131.819151] Lustre: Skipped 1 previous similar message [ 4132.496313] Lustre: server umount lustre-MDT0000 complete [ 4134.475924] LustreError: 186953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907056 with bad export cookie 9883832387271520952 [ 4134.479649] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4134.482092] LustreError: 186953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4134.678498] Lustre: server umount lustre-MDT0001 complete [ 4140.116678] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 4141.497461] Key type lgssc unregistered [ 4141.651534] LNet: 187703:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4141.655761] LNetError: 187703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4141.668755] LNet: Removed LNI 192.168.203.124@tcp [ 4142.113536] Key type .llcrypt unregistered [ 4142.115322] Key type ._llcrypt unregistered [ 4155.271888] Key type ._llcrypt registered [ 4155.273700] Key type .llcrypt registered [ 4155.329388] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4155.753459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4155.779063] alg: No test for adler32 (adler32-zlib) [ 4156.700844] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4156.818308] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 4158.431168] Key type lgssc registered [ 4159.044183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4162.751543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4162.763789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4163.984558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4165.865503] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4170.130024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4170.305458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4172.295168] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4176.161844] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4178.851081] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4182.003657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4182.172835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4184.734210] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4188.268380] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4189.355155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4199.904092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4199.908214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4199.917167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4200.893320] Lustre: lustre-OST0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 4200.930169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.930249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4202.716632] Lustre: server umount lustre-OST0000 complete [ 4206.398515] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8e6e85b4e000.ost_server_uuid 50 [ 4207.235634] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4218.570441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4218.571902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4218.580396] Lustre: Skipped 1 previous similar message [ 4218.583012] Lustre: Skipped 2 previous similar messages [ 4221.664740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4221.668075] Lustre: Skipped 2 previous similar messages [ 4224.810033] Lustre: server umount lustre-MDT0000 complete [ 4226.747248] LustreError: 189079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907148 with bad export cookie 1662578858609529306 [ 4226.749431] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4226.755043] LustreError: 189079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4226.924464] Lustre: server umount lustre-MDT0001 complete [ 4231.323834] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:39:12 (1788907152) [ 4234.755487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4234.982330] LustreError: 193008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4234.993913] LustreError: 193008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4235.027469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4237.174547] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4240.352248] LustreError: 193009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4240.362249] LustreError: 193009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4241.956562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4242.174399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4244.456808] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4248.328558] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4250.983622] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4254.173316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4254.344611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4256.746552] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4259.973439] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4260.523497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4262.992938] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4263.095902] Lustre: lustre-OST0001: new disk, initializing [ 4263.098364] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4263.102595] Lustre: 195846:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4265.673411] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4267.543431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4267.549301] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4267.587210] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4270.303451] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4272.616252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4272.621739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4272.623042] Lustre: Skipped 1 previous similar message [ 4272.628676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4277.578335] Lustre: server umount lustre-OST0000 complete [ 4277.731261] LustreError: 196330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4277.738162] LustreError: 196330:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4280.202451] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4280.290179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4280.748521] LustreError: 195533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4282.241896] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4282.355666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4285.864751] LustreError: 195533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4285.871343] LustreError: 195533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4293.088862] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4293.092261] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.094793] LustreError: Skipped 1 previous similar message [ 4293.095608] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4293.104122] Lustre: Skipped 1 previous similar message [ 4293.106697] Lustre: Skipped 2 previous similar messages [ 4298.210272] LustreError: 196330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4298.220368] LustreError: 196330:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4299.027985] Lustre: server umount lustre-OST0001 complete [ 4303.329329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4303.336670] Lustre: Skipped 1 previous similar message [ 4306.801820] Lustre: server umount lustre-MDT0000 complete [ 4308.742338] LustreError: 192990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907230 with bad export cookie 1662578858609530377 [ 4308.744345] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4308.748225] LustreError: 192990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4308.932701] Lustre: server umount lustre-MDT0001 complete [ 4321.492511] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:40:42 (1788907242) [ 4324.698473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4324.866747] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4324.883961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4324.982130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4324.985351] Lustre: Skipped 1 previous similar message [ 4327.088295] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4331.420649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4331.467605] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4331.495822] Lustre: 199261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4333.640927] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4337.427309] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4340.202910] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4343.501933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4343.606639] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4346.838799] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4348.906884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4351.140907] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4356.451670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4356.554775] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4363.970456] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4374.097310] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4379.618514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4379.633163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4379.669160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4379.672017] Lustre: Skipped 5 previous similar messages [ 4383.447662] Lustre: server umount lustre-OST0000 complete [ 4385.208362] LustreError: 201110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4385.247878] LustreError: 201110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4401.640317] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4401.653623] LustreError: Skipped 1 previous similar message [ 4406.733221] Lustre: server umount lustre-OST0001 complete [ 4411.878594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4411.894218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4412.018823] Lustre: Skipped 4 previous similar messages [ 4412.044662] Lustre: Skipped 6 previous similar messages [ 4417.960355] Lustre: server umount lustre-MDT0000 complete [ 4421.567273] LustreError: 201131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907343 with bad export cookie 1662578858609531763 [ 4421.567613] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4421.581343] LustreError: 201131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4421.940063] Lustre: server umount lustre-MDT0001 complete [ 4448.986921] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:42:49 (1788907369) [ 4462.996996] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 4470.775776] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4496.240955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4496.453424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4496.491495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4496.549167] Lustre: lustre-MDT0000: new disk, initializing [ 4496.596633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4496.600110] Lustre: Skipped 3 previous similar messages [ 4496.610151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4500.222177] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4508.085983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.149150] Lustre: 207353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4508.187975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4508.192130] Lustre: Skipped 1 previous similar message [ 4508.252366] Lustre: lustre-MDT0001: new disk, initializing [ 4508.339544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4508.348361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4511.713591] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4518.689727] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4522.430982] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4526.879648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4527.073406] Lustre: lustre-OST0000: new disk, initializing [ 4527.077294] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4527.082445] Lustre: 209323:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4531.275903] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4534.815436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4534.828961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4534.888644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4537.505240] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4540.886897] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4541.102584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4544.429633] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4544.642934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4550.112082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4550.116535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4550.137150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4550.147815] Lustre: Skipped 3 previous similar messages [ 4560.351138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4560.464106] Lustre: server umount lustre-OST0000 complete [ 4569.925612] Lustre: server umount lustre-MDT0000 complete [ 4570.593037] LustreError: 208916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4570.610593] LustreError: 208916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4573.371820] LustreError: 207346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907495 with bad export cookie 1662578858609533310 [ 4573.374639] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4573.380048] LustreError: 207346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4573.637437] Lustre: server umount lustre-MDT0001 complete [ 4580.862182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4581.211799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4581.217464] Lustre: Skipped 2 previous similar messages [ 4585.007740] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4592.068939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4596.028319] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4602.928136] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4607.698188] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4613.156991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4618.606930] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4625.584633] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4629.662222] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4629.832059] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4634.007808] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4634.172221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4636.128595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4636.142882] LustreError: Skipped 1 previous similar message [ 4637.954491] Lustre: server umount lustre-OST0000 complete [ 4643.183254] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4646.844682] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4673.378322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4676.711202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4676.723931] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4676.724104] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4676.741883] Lustre: Skipped 1 previous similar message [ 4677.702320] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4681.993355] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4692.277808] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 9 sec [ 4697.572902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.574055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4697.586933] Lustre: Skipped 6 previous similar messages [ 4697.590379] Lustre: Skipped 9 previous similar messages [ 4700.196448] Lustre: server umount lustre-OST0000 complete [ 4708.273215] Lustre: server umount lustre-MDT0000 complete [ 4710.212644] LustreError: 211519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907632 with bad export cookie 1662578858609534192 [ 4710.215848] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4710.221209] LustreError: 211519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4710.410091] Lustre: server umount lustre-MDT0001 complete [ 4715.003945] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:47:16 (1788907636) [ 4718.628035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4718.887411] LustreError: 217506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4718.901405] LustreError: 217506:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4718.944198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4718.950772] Lustre: Skipped 3 previous similar messages [ 4721.656963] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4726.639436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4729.021606] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4732.643451] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4735.013494] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4737.835210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4740.157075] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4742.721736] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4744.179283] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4744.254207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4745.733895] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4752.971541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 7 sec [ 4755.602396] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4755.681713] Lustre: lustre-OST0001: new disk, initializing [ 4755.684559] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4755.687905] Lustre: 220924:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4757.961219] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4762.091647] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4763.159851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4763.164709] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4763.194477] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4763.742204] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4763.834809] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4765.471373] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4765.554807] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4768.226549] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4768.229976] LustreError: Skipped 3 previous similar messages [ 4772.422593] Lustre: server umount lustre-OST0001 complete [ 4774.526423] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4774.604506] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4776.162800] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4776.255205] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4800.509499] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4802.889407] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4805.606667] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4805.611931] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4805.612639] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4805.621254] Lustre: Skipped 1 previous similar message [ 4805.671032] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4806.226820] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4808.955922] Lustre: server umount lustre-OST0001 complete [ 4827.352365] Lustre: server umount lustre-OST0000 complete [ 4834.968436] Lustre: server umount lustre-MDT0000 complete [ 4836.385125] LustreError: 223279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907758 with bad export cookie 1662578858609535158 [ 4836.389465] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4836.390851] LustreError: 223279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4842.572840] Lustre: server umount lustre-MDT0001 complete [ 4853.746104] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:49:34 (1788907774) [ 4856.439048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4856.586064] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4856.602835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4858.409542] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4861.831852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4861.898476] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4861.915310] Lustre: 226153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4863.760447] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4866.842668] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4869.105333] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4871.857982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4871.936376] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4874.324659] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4877.446805] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4882.116954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4882.168514] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4884.956742] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4888.421456] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4889.095692] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4890.821756] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4890.947023] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4892.766505] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4892.858413] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4895.191075] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e6e9813d800.ost_server_uuid 50 [ 4895.811659] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e6e9813d800.ost_server_uuid in FULL state after 0 sec [ 4896.214288] Lustre: Permanently deactivating lustre-OST0001 [ 4896.223814] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4897.465937] Lustre: Permanently reactivating lustre-OST0001 [ 4898.788159] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4898.792895] LustreError: Skipped 6 previous similar messages [ 4904.829523] Lustre: server umount lustre-OST0001 complete [ 4909.033012] LustreError: 227996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4909.040228] LustreError: 227996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 29 previous similar messages [ 4912.535363] Lustre: server umount lustre-OST0000 complete [ 4920.287859] Lustre: server umount lustre-MDT0000 complete [ 4921.788551] LustreError: 226901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907844 with bad export cookie 1662578858609536446 [ 4921.790536] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4921.794831] LustreError: 226901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4921.926804] Lustre: server umount lustre-MDT0001 complete [ 4932.446644] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 18:50:53 (1788907853) [ 4935.556209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4935.726228] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4935.743342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4935.749323] Lustre: Skipped 5 previous similar messages [ 4937.699471] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4941.048981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4941.091289] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4941.102810] Lustre: 232835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4942.692658] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4945.481939] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4947.528407] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4950.108139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4950.206684] Lustre: Permanently deactivating lustre-OST0000 [ 4950.212366] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4952.117895] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4952.506566] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4955.220761] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4958.093274] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4958.349943] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4960.823811] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4963.930467] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4964.690714] Lustre: Permanently reactivating lustre-OST0000 [ 4968.011627] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4968.051107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4977.123424] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4977.127581] Lustre: Skipped 16 previous similar messages [ 4977.130296] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4977.133091] Lustre: Skipped 25 previous similar messages [ 4980.309914] Lustre: server umount lustre-OST0001 complete [ 4996.063083] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4996.121981] Lustre: server umount lustre-OST0000 complete [ 5003.681787] Lustre: server umount lustre-MDT0000 complete [ 5005.266500] LustreError: 233580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907927 with bad export cookie 1662578858609538084 [ 5005.267445] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5005.274283] LustreError: 233580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5005.408561] Lustre: server umount lustre-MDT0001 complete [ 5009.895162] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5011.075274] Key type lgssc unregistered [ 5011.216620] LNet: 237695:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5011.220689] LNetError: 237695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5011.236600] LNet: Removed LNI 192.168.203.124@tcp [ 5011.601313] Key type .llcrypt unregistered [ 5011.603173] Key type ._llcrypt unregistered [ 5021.516291] Key type ._llcrypt registered [ 5021.517914] Key type .llcrypt registered [ 5021.559495] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5021.974313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5021.995621] alg: No test for adler32 (adler32-zlib) [ 5022.869354] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5022.971253] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5024.559185] Key type lgssc registered [ 5025.053285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5029.110452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5029.117556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5030.287541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5031.775148] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5034.879257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5034.939153] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5034.955939] Lustre: 239127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5034.962078] Lustre: Permanently deactivating lustre-MDT0001 [ 5034.971785] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5035.126439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5036.726807] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5037.677685] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5037.681198] LustreError: 239168:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5037.687113] LustreError: 239168:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5037.691874] LustreError: 239168:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 5039.555021] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5041.796023] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5044.583842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5044.770281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5045.794930] LustreError: 240979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5045.798662] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5045.803113] LustreError: 240979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5047.170738] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5050.164297] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5050.850152] LustreError: 241121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5053.219118] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5053.346982] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5055.406017] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5055.407498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5055.415164] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5056.034096] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5059.165105] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5060.913983] Lustre: Permanently reactivating lustre-MDT0001 [ 5060.920890] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5060.923760] Lustre: Skipped 2 previous similar messages [ 5065.658036] Lustre: lustre-MDT0001: Client 39af99a1-2dff-4b6c-a7a3-3fbd9f9d7aa7 (at 192.168.203.24@tcp) reconnecting [ 5068.162914] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5068.777231] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5070.818939] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5070.905715] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5072.418811] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5072.503330] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5073.495166] Lustre: Permanently deactivating lustre-MDT0001 [ 5073.503445] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5073.506905] Lustre: Skipped 2 previous similar messages [ 5079.056343] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5085.154385] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5085.157843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5085.164175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5087.203969] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5087.204061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5087.210073] Lustre: Skipped 2 previous similar messages [ 5087.212555] Lustre: Skipped 2 previous similar messages [ 5088.804889] Lustre: server umount lustre-MDT0000 complete [ 5090.239612] LustreError: 239877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908012 with bad export cookie 11812477743959162493 [ 5090.244165] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5090.244891] LustreError: 239877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5090.365586] Lustre: server umount lustre-MDT0001 complete [ 5097.851254] Lustre: server umount lustre-OST0000 complete [ 5101.733935] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:53:42 (1788908022) [ 5105.521047] Lustre: server umount lustre-OST0001 complete [ 5107.866319] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 5111.266779] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5124.344071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5124.436123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5124.440574] Lustre: Skipped 2 previous similar messages [ 5124.450941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5124.490451] Lustre: lustre-MDT0000: new disk, initializing [ 5124.517894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5124.525466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5126.085602] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5130.601757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5130.647710] Lustre: 247430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5130.673096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5130.675266] Lustre: Skipped 1 previous similar message [ 5130.717089] Lustre: lustre-MDT0001: new disk, initializing [ 5130.765126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5130.769779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5132.287498] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5136.434324] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5138.539667] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5141.196597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5141.309727] Lustre: lustre-OST0000: new disk, initializing [ 5141.312216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5141.315941] Lustre: 249400:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5141.351764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5141.355971] Lustre: Skipped 1 previous similar message [ 5143.734679] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5146.102655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5146.106813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5146.142198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5148.104896] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5148.887139] LustreError: 247436:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5153.615173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5153.701210] Lustre: lustre-OST0001: new disk, initializing [ 5153.703815] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5153.708285] Lustre: 250513:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5155.933635] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5158.991222] LustreError: 247436:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5159.002637] LustreError: 248231:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5160.050436] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5160.616529] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5163.018497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5163.022565] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5163.053493] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5169.095107] LustreError: 248231:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5173.217476] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5173.219907] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5173.220527] LustreError: Skipped 1 previous similar message [ 5173.226413] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5173.229475] Lustre: Skipped 1 previous similar message [ 5178.338355] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5178.341339] Lustre: Skipped 1 previous similar message [ 5183.460763] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5183.463579] Lustre: Skipped 1 previous similar message [ 5183.967081] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5184.010767] Lustre: server umount lustre-OST0001 complete [ 5188.579637] LustreError: 249924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5188.587375] LustreError: 249924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5193.701075] LustreError: 249384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5193.707240] LustreError: 249384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5197.280984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5197.282394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.285133] LustreError: Skipped 1 previous similar message [ 5197.290129] Lustre: Skipped 1 previous similar message [ 5197.293763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5197.298913] Lustre: Skipped 1 previous similar message [ 5198.818209] LustreError: 249382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5198.826047] LustreError: 249382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5202.464691] Lustre: server umount lustre-OST0000 complete [ 5203.937867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5214.177015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5214.181940] Lustre: Skipped 7 previous similar messages [ 5218.271174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5218.346210] Lustre: server umount lustre-MDT0000 complete [ 5219.296695] LustreError: 247440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5219.304684] LustreError: 247440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5219.923473] LustreError: 247422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908142 with bad export cookie 11812477743959165300 [ 5219.927277] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5219.929042] LustreError: 247422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5226.105386] Lustre: server umount lustre-MDT0001 complete [ 5230.357030] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5231.486444] Key type lgssc unregistered [ 5231.627929] LNet: 252544:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5231.631653] LNetError: 252544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5231.642324] LNet: Removed LNI 192.168.203.124@tcp [ 5231.993059] Key type .llcrypt unregistered [ 5231.995668] Key type ._llcrypt unregistered [ 5248.156708] Key type ._llcrypt registered [ 5248.158304] Key type .llcrypt registered [ 5248.203508] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5248.753596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5248.773578] alg: No test for adler32 (adler32-zlib) [ 5249.664196] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5249.767702] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5251.359183] Key type lgssc registered [ 5251.876761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5255.215360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5255.226051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5256.352735] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5256.363960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5256.434806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5258.088780] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5261.637883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5261.692525] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5261.708342] Lustre: 254727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5261.828404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5263.568098] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5266.560187] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5268.786476] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5271.693733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5271.797138] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5271.880349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5274.089582] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5276.721429] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5281.300495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5322.210613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5322.211587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5322.213481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5322.214078] LustreError: Skipped 1 previous similar message [ 5322.218086] Lustre: Skipped 1 previous similar message [ 5327.331369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5327.333427] Lustre: Skipped 2 previous similar messages [ 5327.575853] Lustre: server umount lustre-OST0000 complete [ 5329.026942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5333.341579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5333.470184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5335.456250] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5337.867895] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5351.847320] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.852741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.853303] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5351.862818] Lustre: Skipped 1 previous similar message [ 5372.385063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5372.386963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5372.387940] LustreError: Skipped 1 previous similar message [ 5372.388787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5372.395307] Lustre: Skipped 1 previous similar message [ 5377.050484] Lustre: server umount lustre-OST0000 complete [ 5379.552624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5379.556644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5379.563913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5379.566913] Lustre: Skipped 1 previous similar message [ 5382.625903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5384.486311] Lustre: server umount lustre-MDT0000 complete [ 5385.904678] LustreError: 254721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908308 with bad export cookie 13532872926699417176 [ 5385.908848] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5385.910039] LustreError: 254721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5386.050338] Lustre: server umount lustre-MDT0001 complete [ 5390.198710] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5391.368933] Key type lgssc unregistered [ 5391.511592] LNet: 260141:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5391.515932] LNetError: 260141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5391.525845] LNet: Removed LNI 192.168.203.124@tcp [ 5391.903608] Key type .llcrypt unregistered [ 5391.904975] Key type ._llcrypt unregistered [ 5402.274969] Key type ._llcrypt registered [ 5402.276084] Key type .llcrypt registered [ 5402.322513] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5402.615366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5402.622755] alg: No test for adler32 (adler32-zlib) [ 5403.492228] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5403.597200] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5405.191197] Key type lgssc registered [ 5405.608756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5408.476252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5408.482351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5409.630801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5411.051191] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5414.238076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5414.383758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5415.807737] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5418.448796] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5420.439588] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5423.261860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5423.413553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5425.608536] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5428.417727] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5434.665597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5445.089387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5445.089652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5445.091931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5445.094163] LustreError: Skipped 1 previous similar message [ 5445.100272] Lustre: Skipped 1 previous similar message [ 5448.859022] Lustre: server umount lustre-OST0000 complete [ 5455.329240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5455.329560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5455.330933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5455.330939] Lustre: Skipped 1 previous similar message [ 5455.344100] Lustre: Skipped 1 previous similar message [ 5456.348960] Lustre: server umount lustre-MDT0000 complete [ 5457.668047] LustreError: 263373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908379 with bad export cookie 14507684071645503529 [ 5457.670513] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5457.673240] LustreError: 263373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5457.794023] Lustre: server umount lustre-MDT0001 complete [ 5461.589311] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5462.685541] Key type lgssc unregistered [ 5462.807202] LNet: 265831:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5462.810291] LNetError: 265831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5462.822295] LNet: Removed LNI 192.168.203.124@tcp [ 5463.128275] Key type .llcrypt unregistered [ 5463.129418] Key type ._llcrypt unregistered [ 5470.261277] Key type ._llcrypt registered [ 5470.262660] Key type .llcrypt registered [ 5470.302411] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5470.675923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5470.723631] alg: No test for adler32 (adler32-zlib) [ 5471.587549] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5471.673982] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5473.255228] Key type lgssc registered [ 5473.690450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5476.235873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5476.243472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5477.383907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5478.780464] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5481.621952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5481.754612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5483.099435] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5485.559154] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5487.510220] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5489.878617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5489.992499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5491.946790] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5494.426278] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5502.310430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5511.648213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5511.651960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5511.658121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5512.673531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5512.674847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5512.676570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5514.638682] Lustre: server umount lustre-OST0000 complete [ 5517.793099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5517.793350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5517.794679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5517.794688] Lustre: Skipped 1 previous similar message [ 5517.798466] Lustre: Skipped 1 previous similar message [ 5522.074553] Lustre: server umount lustre-MDT0000 complete [ 5522.911905] LustreError: 267030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5522.917574] LustreError: 267030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5523.281273] LustreError: 267057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908445 with bad export cookie 15529074726429204382 [ 5523.284775] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5523.286378] LustreError: 267057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5523.407055] Lustre: server umount lustre-MDT0001 complete [ 5527.113320] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5528.184894] Key type lgssc unregistered [ 5528.306320] LNet: 270726:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5528.308643] LNetError: 270726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5528.314314] LNet: Removed LNI 192.168.203.124@tcp [ 5528.642479] Key type .llcrypt unregistered [ 5528.643537] Key type ._llcrypt unregistered [ 5538.149050] Key type ._llcrypt registered [ 5538.150716] Key type .llcrypt registered [ 5538.193551] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5538.572108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5538.599366] alg: No test for adler32 (adler32-zlib) [ 5539.462411] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5539.552433] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5541.143152] Key type lgssc registered [ 5541.512208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5544.062567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5544.069330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5545.204102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5546.453456] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5549.199837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5549.322574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5550.713471] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5553.331410] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5555.206819] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5557.526294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5557.652623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5559.728456] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5562.596507] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5569.961300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5585.381918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5585.382254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5585.388991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5585.395423] Lustre: Skipped 1 previous similar message [ 5587.856711] Lustre: server umount lustre-OST0000 complete [ 5590.497599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5590.497839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5590.499137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5590.501966] LustreError: Skipped 1 previous similar message [ 5590.507297] Lustre: Skipped 2 previous similar messages [ 5595.297729] Lustre: server umount lustre-MDT0000 complete [ 5595.616724] LustreError: 272147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5595.625122] LustreError: 272147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5596.717798] LustreError: 272109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908518 with bad export cookie 7105922603215662548 [ 5596.720472] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5596.727251] LustreError: 272109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5596.841592] Lustre: server umount lustre-MDT0001 complete [ 5600.986521] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5602.233304] Key type lgssc unregistered [ 5602.382602] LNet: 276450:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5602.387184] LNetError: 276450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5602.396874] LNet: Removed LNI 192.168.203.124@tcp [ 5602.759913] Key type .llcrypt unregistered [ 5602.761390] Key type ._llcrypt unregistered [ 5610.379057] Key type ._llcrypt registered [ 5610.380502] Key type .llcrypt registered [ 5610.424605] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5610.766563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5610.822756] alg: No test for adler32 (adler32-zlib) [ 5611.722313] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5611.828295] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5613.431148] Key type lgssc registered [ 5613.855406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5616.739330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5616.746679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5617.934903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5619.502663] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5623.036847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.237247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5624.911182] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5627.901488] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5630.253102] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5633.053970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5633.216755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5635.539052] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5638.575260] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5643.496281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5653.988970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5653.997183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5653.997707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5654.002358] Lustre: Skipped 1 previous similar message [ 5658.323540] Lustre: server umount lustre-OST0000 complete [ 5664.227937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5664.233881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5664.234026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5664.237714] Lustre: Skipped 1 previous similar message [ 5664.259895] Lustre: Skipped 2 previous similar messages [ 5667.575964] Lustre: server umount lustre-MDT0000 complete [ 5669.346416] LustreError: 277663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5669.357465] LustreError: 277663:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5670.698821] LustreError: 277631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908592 with bad export cookie 1827870764646093873 [ 5670.701615] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5670.703384] LustreError: 277631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5670.967028] Lustre: server umount lustre-MDT0001 complete [ 5679.243125] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5681.946207] Key type lgssc unregistered [ 5682.212916] LNet: 281362:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5682.217376] LNetError: 281362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5682.235334] LNet: Removed LNI 192.168.203.124@tcp [ 5683.055319] Key type .llcrypt unregistered [ 5683.060241] Key type ._llcrypt unregistered [ 5694.878509] Key type ._llcrypt registered [ 5694.882904] Key type .llcrypt registered [ 5695.031616] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5710.131319] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 5719.647881] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5720.270117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5720.353793] alg: No test for adler32 (adler32-zlib) [ 5721.337353] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5721.527319] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5723.135171] Key type lgssc registered [ 5723.915518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5743.841925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5743.857573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5744.973859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5744.997270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5745.047149] Lustre: lustre-MDT0000: new disk, initializing [ 5745.091049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5745.103362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5747.045926] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5753.365981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5753.445095] Lustre: 284893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5753.482660] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5753.485788] Lustre: Skipped 1 previous similar message [ 5753.545181] Lustre: lustre-MDT0001: new disk, initializing [ 5753.613969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5753.638836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5753.643612] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5756.235609] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5762.164543] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5765.245700] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5769.309396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5769.472060] Lustre: lustre-OST0000: new disk, initializing [ 5769.475164] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5769.479182] Lustre: 286856:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5769.543795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5773.120926] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5777.953228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5777.959859] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5778.034216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5779.244878] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5781.934675] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5782.092067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5784.485595] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5784.602604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5788.132248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5788.133696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5788.144333] Lustre: Skipped 1 previous similar message [ 5788.151946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5791.869682] Lustre: server umount lustre-OST0000 complete [ 5794.785089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5794.789920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5794.798739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5794.802000] Lustre: Skipped 1 previous similar message [ 5798.369560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5798.369636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5800.136291] Lustre: server umount lustre-MDT0000 complete [ 5802.054149] LustreError: 284885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908724 with bad export cookie 11479801887393774948 [ 5802.060629] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5802.062387] LustreError: 284885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5802.294604] Lustre: server umount lustre-MDT0001 complete [ 5806.716756] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:05:27 (1788908727) [ 5810.795490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5811.065557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5813.179678] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5816.291677] LustreError: 289270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5816.302861] LustreError: 289270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5818.284735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5818.604877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5821.411764] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5825.758500] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5828.966522] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5832.944607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5833.141955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5836.471585] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5841.199960] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5848.545318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5848.547400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5848.557110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5848.569039] Lustre: Skipped 2 previous similar messages [ 5853.675271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5853.679024] Lustre: Skipped 1 previous similar message [ 5854.729138] Lustre: server umount lustre-OST0000 complete [ 5858.793729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.804027] Lustre: Skipped 2 previous similar messages [ 5859.303890] Lustre: server umount lustre-MDT0000 complete [ 5861.819923] LustreError: 289249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908784 with bad export cookie 11479801887393775837 [ 5861.826982] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5861.827715] LustreError: 289249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5862.054438] Lustre: server umount lustre-MDT0001 complete [ 5868.669310] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5870.211394] Key type lgssc unregistered [ 5870.435678] LNet: 292800:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5870.444951] LNetError: 292800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5870.457039] LNet: Removed LNI 192.168.203.124@tcp [ 5871.004813] Key type .llcrypt unregistered [ 5871.007118] Key type ._llcrypt unregistered [ 5886.476376] Key type ._llcrypt registered [ 5886.479175] Key type .llcrypt registered [ 5886.579205] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 5896.317373] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5896.974857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5896.987187] alg: No test for adler32 (adler32-zlib) [ 5897.940734] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5898.106420] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5899.735154] Key type lgssc registered [ 5900.419759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5927.890984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5927.903365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5929.048799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5929.076582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5929.128606] Lustre: lustre-MDT0000: new disk, initializing [ 5929.172053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5929.188501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5931.555739] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5938.478604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5938.557789] Lustre: 296601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5938.597466] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5938.601710] Lustre: Skipped 1 previous similar message [ 5938.654356] Lustre: lustre-MDT0001: new disk, initializing [ 5938.710990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5938.740102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5938.753215] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5941.798098] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5948.171598] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5951.704434] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5956.037108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5956.205065] Lustre: lustre-OST2710: new disk, initializing [ 5956.208193] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5956.213926] Lustre: 298568:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5956.283219] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5959.771772] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5964.315936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5964.323864] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5964.374473] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5965.510070] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5969.099264] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5969.210830] Lustre: lustre-OST03e8: new disk, initializing [ 5969.213124] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5969.216402] Lustre: 299631:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5969.263370] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5972.372656] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5974.552740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5974.559587] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5974.605453] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5975.964367] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5980.830955] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5980.954290] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5983.120584] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5983.232545] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5985.527373] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5985.625278] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5988.011246] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5988.141490] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5994.978580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5994.987167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5994.989685] Lustre: Skipped 2 previous similar messages [ 5994.998675] Lustre: Skipped 2 previous similar messages [ 5994.999815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5996.797045] Lustre: server umount lustre-MDT0000 complete [ 5998.979433] LustreError: 296591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908921 with bad export cookie 13960037405941165115 [ 5998.980996] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5998.986899] LustreError: 296591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5999.215207] Lustre: server umount lustre-MDT0001 complete [ 6010.835522] Lustre: server umount lustre-OST2710 complete [ 6023.292099] Lustre: server umount lustre-OST03e8 complete [ 6034.580651] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 6040.201448] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6060.390442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6060.552580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6060.578388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6060.638383] Lustre: lustre-MDT0000: new disk, initializing [ 6060.687203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6060.698157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6063.041480] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6070.412245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6070.472152] Lustre: 304873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6070.541268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6070.545321] Lustre: Skipped 1 previous similar message [ 6070.597393] Lustre: lustre-MDT0001: new disk, initializing [ 6070.654419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6070.688993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6070.697484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6073.079874] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6079.305779] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6082.654624] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6086.814841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6086.996559] Lustre: lustre-OST0000: new disk, initializing [ 6086.999632] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6087.004878] Lustre: 306841:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6087.062132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6090.191067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6090.199459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6090.281746] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6090.757588] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6097.232541] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6100.191216] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6100.371867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6103.119370] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6103.265854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6105.579750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6105.580716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6105.586819] LustreError: Skipped 1 previous similar message [ 6105.600827] Lustre: Skipped 1 previous similar message [ 6105.607634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6110.716423] Lustre: server umount lustre-OST0000 complete [ 6115.808539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6115.810644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6115.816887] Lustre: Skipped 1 previous similar message [ 6115.825767] Lustre: Skipped 2 previous similar messages [ 6119.048760] Lustre: server umount lustre-MDT0000 complete [ 6120.933575] LustreError: 304885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6120.946513] LustreError: 304885:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6121.015311] LustreError: 304866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909043 with bad export cookie 13960037405941166781 [ 6121.016925] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6121.020117] LustreError: 304866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6121.194489] Lustre: server umount lustre-MDT0001 complete [ 6125.255617] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:10:46 (1788909046) [ 6126.063917] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6126.984442] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:10:47 (1788909047) [ 6128.570490] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6142.072928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6142.229344] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6142.247528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6142.338238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6144.441142] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6148.398485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6148.457299] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6148.478055] Lustre: 310694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6150.242955] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6167.014468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6167.116949] Lustre: Denying initial registration attempt from nid 192.168.203.124@tcp, specified as failover [ 6167.122384] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6167.129610] LustreError: 312145:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -99 [ 6167.136439] LustreError: 312145:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-OST0000 [ 6167.139211] LustreError: 312145:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6167.181857] Lustre: server umount lustre-OST0000 complete [ 6167.184391] LustreError: 312145:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6172.129360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6172.131249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6172.136024] Lustre: Skipped 1 previous similar message [ 6176.379671] Lustre: server umount lustre-MDT0000 complete [ 6177.248341] LustreError: 311439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6177.256304] LustreError: 311439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6178.133895] LustreError: 310685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909100 with bad export cookie 13960037405941167663 [ 6178.135796] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6178.138991] LustreError: 310685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6178.283092] Lustre: server umount lustre-MDT0001 complete [ 6183.157940] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6184.446871] Key type lgssc unregistered [ 6184.589129] LNet: 313265:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6184.593869] LNetError: 313265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6184.606087] LNet: Removed LNI 192.168.203.124@tcp [ 6185.020960] Key type .llcrypt unregistered [ 6185.024451] Key type ._llcrypt unregistered [ 6190.111430] Key type ._llcrypt registered [ 6190.112982] Key type .llcrypt registered [ 6190.172239] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6190.579278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6190.588738] alg: No test for adler32 (adler32-zlib) [ 6191.466077] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6191.580087] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6193.183575] Key type lgssc registered [ 6193.741364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6222.377316] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6226.137927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6226.151139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6227.264903] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6227.279115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6227.350189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6229.294608] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6233.482851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6233.549672] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6233.575849] Lustre: 316006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6233.726519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6235.575933] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6238.823526] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6241.318115] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6257.896472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6258.007090] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6258.091140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6260.589431] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6263.674433] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6269.412348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6269.413719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6269.413776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6269.417349] Lustre: Skipped 1 previous similar message [ 6269.424507] Lustre: Skipped 1 previous similar message [ 6270.687852] Lustre: server umount lustre-OST0000 complete [ 6274.529363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6274.530577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6274.535957] LustreError: Skipped 1 previous similar message [ 6274.543325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6274.550705] Lustre: Skipped 1 previous similar message [ 6278.526436] Lustre: server umount lustre-MDT0000 complete [ 6280.234908] LustreError: 317870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909202 with bad export cookie 12083703604553753893 [ 6280.239694] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6280.243988] LustreError: 317870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6280.409249] Lustre: server umount lustre-MDT0001 complete [ 6285.759573] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6287.107868] Key type lgssc unregistered [ 6287.252659] LNet: 319579:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6287.256316] LNetError: 319579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6287.268594] LNet: Removed LNI 192.168.203.124@tcp [ 6287.624440] Key type .llcrypt unregistered [ 6287.626324] Key type ._llcrypt unregistered [ 6297.125479] Key type ._llcrypt registered [ 6297.127146] Key type .llcrypt registered [ 6297.177797] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6297.305380] Key type .llcrypt unregistered [ 6297.307078] Key type ._llcrypt unregistered [ 6305.090341] Key type ._llcrypt registered [ 6305.091658] Key type .llcrypt registered [ 6305.139149] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6305.597872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6305.605763] alg: No test for adler32 (adler32-zlib) [ 6306.475483] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6306.582395] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6308.175219] Key type lgssc registered [ 6308.649101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6327.944909] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing lnet_if_list [ 6328.428114] LNet: Added LNI 192.168.203.124@tcp1 [8/256/0/180] [ 6332.646400] loop0: detected capacity change from 0 to 204800000 [ 6332.721539] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6332.727394] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6332.733070] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6346.319580] loop0: detected capacity change from 0 to 204800000 [ 6346.391929] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6346.397697] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6346.405359] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6360.676451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6360.684693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6361.816684] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6361.828369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6361.920646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6363.649703] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6367.594377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6367.650615] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6367.671156] Lustre: 323476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6367.813616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6369.687149] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6373.118305] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6375.838923] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6378.298937] loop0: detected capacity change from 0 to 204800000 [ 6394.276211] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6394.405330] Lustre: sv10053-OST0000: new disk, initializing [ 6394.409096] Lustre: srv-sv10053-OST0000: No data found on store. Initialize space. [ 6394.413594] Lustre: 325553:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6394.447935] Lustre: sv10053-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6397.153306] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6402.428389] loop1: detected capacity change from 0 to 204800000 [ 6418.279877] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6418.346396] Lustre: sv10053-OST0001: new disk, initializing [ 6418.349156] Lustre: srv-sv10053-OST0001: No data found on store. Initialize space. [ 6418.354257] Lustre: 326669:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6418.387036] Lustre: sv10053-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6420.980429] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6427.513614] Lustre: server umount sv10053-OST0001 complete [ 6432.290879] Lustre: server umount sv10053-OST0000 complete [ 6439.394745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6439.401650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6439.409302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6442.976911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6442.977418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6442.992581] Lustre: Skipped 1 previous similar message [ 6448.096493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6448.106844] Lustre: Skipped 1 previous similar message [ 6453.215776] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6453.216875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6453.227701] Lustre: Skipped 1 previous similar message [ 6453.298707] Lustre: server umount lustre-MDT0000 complete [ 6455.167380] LustreError: 323468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909377 with bad export cookie 8658567620447897405 [ 6455.170347] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6455.172604] LustreError: 323468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6455.383433] Lustre: server umount lustre-MDT0001 complete [ 6461.274277] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6462.529482] Key type lgssc unregistered [ 6462.674430] LNet: 329418:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6462.679034] LNetError: 329418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6462.684250] LNet: Removed LNI 192.168.203.124@tcp [ 6463.718065] LNet: Removed LNI 192.168.203.124@tcp1 [ 6464.100158] Key type .llcrypt unregistered [ 6464.101389] Key type ._llcrypt unregistered [ 6471.875229] Key type ._llcrypt registered [ 6471.877094] Key type .llcrypt registered [ 6471.923354] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6472.064256] Key type .llcrypt unregistered [ 6472.066308] Key type ._llcrypt unregistered [ 6480.282431] Key type ._llcrypt registered [ 6480.284119] Key type .llcrypt registered [ 6480.334862] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6480.798564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6480.874466] alg: No test for adler32 (adler32-zlib) [ 6481.777728] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6481.905802] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6483.527347] Key type lgssc registered [ 6484.098668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6494.178384] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing lnet_if_list [ 6494.692214] LNet: Added LNI 192.168.203.124@tcp1 [8/256/0/180] [ 6498.652889] loop0: detected capacity change from 0 to 204800000 [ 6498.769805] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6498.776122] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6498.784356] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6512.841494] loop0: detected capacity change from 0 to 204800000 [ 6512.924368] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6512.935380] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6512.941254] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6527.379749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6527.393278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6528.548495] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6528.568867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6528.665340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6530.422919] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6534.316281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6534.368815] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6534.385082] Lustre: 333213:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6534.507111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6536.326074] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6540.064209] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6542.972173] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6545.579116] loop0: detected capacity change from 0 to 204800000 [ 6560.993935] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6561.101954] Lustre: sv5112-OST0000: new disk, initializing [ 6561.106358] Lustre: srv-sv5112-OST0000: No data found on store. Initialize space. [ 6561.110543] Lustre: 335291:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6561.141915] Lustre: sv5112-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6563.724461] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6568.855774] loop1: detected capacity change from 0 to 204800000 [ 6584.675859] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6584.755575] Lustre: sv5112-OST0001: new disk, initializing [ 6584.758351] Lustre: srv-sv5112-OST0001: No data found on store. Initialize space. [ 6584.762958] Lustre: 336408:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6584.794364] Lustre: sv5112-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6587.180278] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6593.396229] Lustre: server umount sv5112-OST0001 complete [ 6597.091933] Lustre: server umount sv5112-OST0000 complete [ 6602.721283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6602.722546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6602.727501] Lustre: Skipped 1 previous similar message [ 6607.840023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6607.844357] Lustre: Skipped 2 previous similar messages [ 6608.042283] Lustre: server umount lustre-MDT0000 complete [ 6609.639994] LustreError: 333204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909531 with bad export cookie 7376479789827209136 [ 6609.644503] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6609.648236] LustreError: 333204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6609.801130] Lustre: server umount lustre-MDT0001 complete [ 6615.419630] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6616.689381] Key type lgssc unregistered [ 6616.823765] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.24@tcp [ 6616.842650] LNet: 339158:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6616.850990] LNetError: 339158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6616.857436] LNet: Removed LNI 192.168.203.124@tcp [ 6617.893788] LNet: Removed LNI 192.168.203.124@tcp1 [ 6618.352711] Key type .llcrypt unregistered [ 6618.353951] Key type ._llcrypt unregistered [ 6626.157360] Key type ._llcrypt registered [ 6626.158917] Key type .llcrypt registered [ 6626.210754] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6626.336395] Key type .llcrypt unregistered [ 6626.338030] Key type ._llcrypt unregistered [ 6634.328930] Key type ._llcrypt registered [ 6634.330641] Key type .llcrypt registered [ 6634.385370] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6634.845353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6634.901623] alg: No test for adler32 (adler32-zlib) [ 6635.789186] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6635.901733] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6637.488534] Key type lgssc registered [ 6637.940228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6646.512813] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing lnet_if_list [ 6647.058412] LNet: Added LNI 192.168.203.124@tcp1 [8/256/0/180] [ 6651.238411] loop0: detected capacity change from 0 to 204800000 [ 6651.364558] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6651.369798] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6651.377746] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6664.832256] loop0: detected capacity change from 0 to 204800000 [ 6664.910435] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6664.915475] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6664.922593] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6679.189440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6679.205127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6680.313662] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6680.325701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6680.398110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6682.098841] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6685.969905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6686.048849] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6686.076403] Lustre: 342957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6686.206739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6688.000748] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6691.374879] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6693.755247] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6695.996412] loop0: detected capacity change from 0 to 204800000 [ 6711.652897] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6711.773236] Lustre: sv18258-OST0000: new disk, initializing [ 6711.776461] Lustre: srv-sv18258-OST0000: No data found on store. Initialize space. [ 6711.783228] Lustre: 345034:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6711.809558] Lustre: sv18258-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6714.592319] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6722.855984] Lustre: server umount sv18258-OST0000 complete [ 6730.720824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6730.721774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6730.727442] Lustre: Skipped 1 previous similar message [ 6730.735742] Lustre: Skipped 1 previous similar message [ 6735.105116] Lustre: server umount lustre-MDT0000 complete [ 6737.037287] LustreError: 342952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909659 with bad export cookie 1034209842169004604 [ 6737.040774] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6737.047402] LustreError: 342952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6737.219611] Lustre: server umount lustre-MDT0001 complete [ 6742.948309] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6744.393435] Key type lgssc unregistered [ 6744.590434] LNet: 347481:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6744.596669] LNetError: 347481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6744.599806] LNet: Removed LNI 192.168.203.124@tcp [ 6745.639910] LNet: Removed LNI 192.168.203.124@tcp1 [ 6746.017762] Key type .llcrypt unregistered [ 6746.020775] Key type ._llcrypt unregistered [ 6753.869457] Key type ._llcrypt registered [ 6753.871165] Key type .llcrypt registered [ 6753.921604] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6754.059440] Key type .llcrypt unregistered [ 6754.061401] Key type ._llcrypt unregistered [ 6762.314776] Key type ._llcrypt registered [ 6762.316677] Key type .llcrypt registered [ 6762.374444] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6762.768562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6762.806062] alg: No test for adler32 (adler32-zlib) [ 6763.706132] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6763.814964] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6765.415474] Key type lgssc registered [ 6765.943193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6776.504779] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing lnet_if_list [ 6777.091594] LNet: Added LNI 192.168.203.124@tcp1 [8/256/0/180] [ 6781.851644] loop0: detected capacity change from 0 to 204800000 [ 6781.933212] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6781.940182] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6781.949296] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6796.020714] loop0: detected capacity change from 0 to 204800000 [ 6796.122238] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6796.132150] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6796.143985] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6810.356905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6810.365988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6811.500747] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6811.514464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6811.607528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6813.542577] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6817.758421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6817.809273] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6817.826754] Lustre: 351323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6817.959630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6819.687614] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6822.698528] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6824.934273] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6826.986565] loop0: detected capacity change from 0 to 204800000 [ 6842.722362] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6842.831794] Lustre: sv14846-OST0000: new disk, initializing [ 6842.835205] Lustre: srv-sv14846-OST0000: No data found on store. Initialize space. [ 6842.839581] Lustre: 353400:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6842.870852] Lustre: sv14846-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6845.614771] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6853.455268] Lustre: server umount sv14846-OST0000 complete [ 6859.232558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6859.236520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6859.244738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6860.768272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6860.769262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6864.609179] Lustre: server umount lustre-MDT0000 complete [ 6866.162692] LustreError: 351318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909788 with bad export cookie 1052879213730851449 [ 6866.167056] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6866.167273] LustreError: 351318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6866.306399] Lustre: server umount lustre-MDT0001 complete [ 6871.601173] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6872.827308] Key type lgssc unregistered [ 6872.956426] LNet: 355850:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6872.959675] LNetError: 355850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6872.962491] LNet: Removed LNI 192.168.203.124@tcp [ 6874.021587] LNet: Removed LNI 192.168.203.124@tcp1 [ 6874.404360] Key type .llcrypt unregistered [ 6874.407857] Key type ._llcrypt unregistered [ 6882.006060] Key type ._llcrypt registered [ 6882.008924] Key type .llcrypt registered [ 6882.063107] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6882.189379] Key type .llcrypt unregistered [ 6882.191417] Key type ._llcrypt unregistered [ 6889.941103] Key type ._llcrypt registered [ 6889.942792] Key type .llcrypt registered [ 6889.992419] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6890.371231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6890.422564] alg: No test for adler32 (adler32-zlib) [ 6891.327863] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 6891.460644] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6893.055170] Key type lgssc registered [ 6893.564141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6902.920254] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing lnet_if_list [ 6903.464427] LNet: Added LNI 192.168.203.124@tcp1 [8/256/0/180] [ 6907.729747] loop0: detected capacity change from 0 to 204800000 [ 6907.815517] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6907.821742] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6907.827383] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6922.417810] loop0: detected capacity change from 0 to 204800000 [ 6922.575705] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6922.583233] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6922.591017] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6936.616435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6936.623910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6937.770670] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6937.791418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6937.882281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6939.895267] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6943.472764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6943.526872] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6943.539979] Lustre: 359661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6943.650519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6945.238844] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6948.205371] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6950.451495] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6952.630866] loop0: detected capacity change from 0 to 204800000 [ 6968.035289] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6968.164167] Lustre: sv30821-OST0000: new disk, initializing [ 6968.166288] Lustre: srv-sv30821-OST0000: No data found on store. Initialize space. [ 6968.169952] Lustre: 361740:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6968.196560] Lustre: sv30821-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6970.547025] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6975.430784] loop1: detected capacity change from 0 to 204800000 [ 6990.631842] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6990.713782] Lustre: sv30821-OST0001: new disk, initializing [ 6990.716433] Lustre: srv-sv30821-OST0001: No data found on store. Initialize space. [ 6990.720491] Lustre: 362905:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6990.756663] Lustre: sv30821-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6993.385365] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6999.543381] Lustre: server umount sv30821-OST0001 complete [ 7003.612585] Lustre: server umount sv30821-OST0000 complete [ 7010.272087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7010.275913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7010.283588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7013.344126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7013.344656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7014.677303] Lustre: server umount lustre-MDT0000 complete [ 7016.104987] LustreError: 359654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788909938 with bad export cookie 16776795286373208671 [ 7016.107334] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7016.110713] LustreError: 359654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7016.235414] Lustre: server umount lustre-MDT0001 complete [ 7021.320779] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7022.510365] Key type lgssc unregistered [ 7022.632720] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.24@tcp [ 7022.652672] LNet: 365654:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7022.655450] LNetError: 365654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7022.658690] LNet: Removed LNI 192.168.203.124@tcp [ 7023.717444] LNet: Removed LNI 192.168.203.124@tcp1 [ 7024.092771] Key type .llcrypt unregistered [ 7024.094519] Key type ._llcrypt unregistered [ 7031.273229] Key type ._llcrypt registered [ 7031.274963] Key type .llcrypt registered [ 7031.323305] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7031.451560] Key type .llcrypt unregistered [ 7031.453630] Key type ._llcrypt unregistered [ 7039.419376] Key type ._llcrypt registered [ 7039.421357] Key type .llcrypt registered [ 7039.478394] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 7039.927667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7039.979359] alg: No test for adler32 (adler32-zlib) [ 7040.915384] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7041.030420] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 7042.631179] Key type lgssc registered [ 7043.149217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7052.025310] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing lnet_if_list [ 7052.507491] LNet: Added LNI 192.168.203.124@tcp1 [8/256/0/180] [ 7065.091469] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7066.369839] Key type lgssc unregistered [ 7066.494584] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.24@tcp [ 7066.518968] LNet: 369586:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7066.522824] LNetError: 369586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7066.526591] LNet: Removed LNI 192.168.203.124@tcp [ 7066.533570] LNet: Removed LNI 192.168.203.124@tcp1 [ 7066.879469] Key type .llcrypt unregistered [ 7066.881522] Key type ._llcrypt unregistered [ 7074.434131] Key type ._llcrypt registered [ 7074.435568] Key type .llcrypt registered [ 7074.486950] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7074.612563] Key type .llcrypt unregistered [ 7074.614480] Key type ._llcrypt unregistered [ 7082.116449] Key type ._llcrypt registered [ 7082.117440] Key type .llcrypt registered [ 7082.158773] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 7082.597800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7082.619602] alg: No test for adler32 (adler32-zlib) [ 7083.498475] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7083.612664] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 7085.215199] Key type lgssc registered [ 7085.604216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7092.530319] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing lnet_if_list [ 7092.970829] LNet: Added LNI 192.168.203.124@tcp1 [8/256/0/180] [ 7093.587537] LNet: Added LNI 192.168.203.124@tcp2 [8/256/0/180] [ 7097.307058] loop0: detected capacity change from 0 to 204800000 [ 7097.364390] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7097.370509] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7097.376919] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7110.517677] loop0: detected capacity change from 0 to 204800000 [ 7110.591225] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7110.596555] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7110.601618] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7124.391341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7124.400899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7125.509198] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7125.517622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7125.581246] LNetError: 370890:0:(peer.c:2149:lnet_peer_add_nid()) Peer 192.168.203.124@tcp NID 192.168.203.124@tcp is already locked with peer 192.168.203.124@tcp [ 7125.587287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7127.079620] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7130.268897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.313719] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7130.326413] Lustre: 373473:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7130.401851] LNetError: 370890:0:(peer.c:2149:lnet_peer_add_nid()) Peer 192.168.203.124@tcp NID 192.168.203.124@tcp is already locked with peer 192.168.203.124@tcp [ 7130.443225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7132.126871] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7134.700783] LNetError: 370890:0:(peer.c:2149:lnet_peer_add_nid()) Peer 192.168.203.124@tcp NID 192.168.203.124@tcp is already locked with peer 192.168.203.124@tcp [ 7134.706243] LNetError: 370890:0:(peer.c:2149:lnet_peer_add_nid()) Skipped 2 previous similar messages [ 7134.988726] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7136.989277] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7138.912342] loop0: detected capacity change from 0 to 204800000 [ 7153.954587] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7154.058099] Lustre: sv2167-OST0000: new disk, initializing [ 7154.061268] Lustre: srv-sv2167-OST0000: No data found on store. Initialize space. [ 7154.064893] Lustre: 375547:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7154.096917] Lustre: sv2167-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7156.674680] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7161.913982] loop1: detected capacity change from 0 to 204800000 [ 7177.056943] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7177.104266] Lustre: sv2167-OST0001: new disk, initializing [ 7177.105770] Lustre: srv-sv2167-OST0001: No data found on store. Initialize space. [ 7177.108781] Lustre: 376716:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7177.131682] Lustre: sv2167-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7179.148170] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7184.814444] Lustre: server umount sv2167-OST0001 complete [ 7188.297332] Lustre: server umount sv2167-OST0000 complete [ 7196.129320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7196.130415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7196.133662] Lustre: Skipped 1 previous similar message [ 7196.137906] Lustre: Skipped 1 previous similar message [ 7201.248274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7201.251407] Lustre: Skipped 1 previous similar message [ 7206.368074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7206.370812] Lustre: Skipped 1 previous similar message [ 7207.391102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7207.462417] Lustre: server umount lustre-MDT0000 complete [ 7208.934662] LustreError: 373466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910131 with bad export cookie 16791653680795103135 [ 7208.936829] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7208.940989] LustreError: 373466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7209.078518] Lustre: server umount lustre-MDT0001 complete [ 7214.260599] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7215.360576] Key type lgssc unregistered [ 7215.489391] LNet: 379467:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7215.492122] LNetError: 379467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7215.494153] LNet: Removed LNI 192.168.203.124@tcp [ 7216.546855] LNet: Removed LNI 192.168.203.124@tcp1 [ 7216.548576] LNetError: 379467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7216.552322] LNetError: 379467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7216.562399] LNet: Removed LNI 192.168.203.124@tcp2 [ 7216.869333] Key type .llcrypt unregistered [ 7216.870565] Key type ._llcrypt unregistered [ 7223.551265] Key type ._llcrypt registered [ 7223.552713] Key type .llcrypt registered [ 7223.589762] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7223.681213] Key type .llcrypt unregistered [ 7223.682520] Key type ._llcrypt unregistered [ 7230.176115] Key type ._llcrypt registered [ 7230.177528] Key type .llcrypt registered [ 7230.210315] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 7230.572695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7230.614796] alg: No test for adler32 (adler32-zlib) [ 7231.464956] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7231.552292] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 7233.127244] Key type lgssc registered [ 7233.504256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7240.729768] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing lnet_if_list [ 7241.155331] LNet: Added LNI 192.168.203.124@tcp1 [8/256/0/180] [ 7244.482042] loop0: detected capacity change from 0 to 204800000 [ 7244.541670] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7244.546577] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7244.551052] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7257.975843] loop0: detected capacity change from 0 to 204800000 [ 7258.046566] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7258.051779] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7258.059906] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7271.643585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7271.650502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7272.770142] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7272.781790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7272.848635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7274.342156] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7277.303036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7277.338901] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7277.350411] Lustre: 383310:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7277.453906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7278.781266] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7281.172580] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7283.067793] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7284.867304] loop0: detected capacity change from 0 to 204800000 [ 7300.000888] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7300.085477] Lustre: sv12780-OST0000: new disk, initializing [ 7300.087544] Lustre: srv-sv12780-OST0000: No data found on store. Initialize space. [ 7300.091393] Lustre: 385386:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7300.108289] Lustre: sv12780-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7301.971218] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7305.778663] loop1: detected capacity change from 0 to 204800000 [ 7320.737977] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7320.785972] Lustre: sv12780-OST0001: new disk, initializing [ 7320.788588] Lustre: srv-sv12780-OST0001: No data found on store. Initialize space. [ 7320.792666] Lustre: 386501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7320.811648] Lustre: sv12780-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7322.894076] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7328.261678] Lustre: server umount sv12780-OST0001 complete [ 7331.981716] Lustre: server umount sv12780-OST0000 complete [ 7338.976119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7338.979533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7338.985260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7341.024538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7341.024576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7341.032400] Lustre: Skipped 1 previous similar message [ 7342.564638] Lustre: server umount lustre-MDT0000 complete [ 7344.009363] LustreError: 383301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910266 with bad export cookie 2110148250114863204 [ 7344.011860] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7344.014852] LustreError: 383301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7344.138653] Lustre: server umount lustre-MDT0001 complete [ 7349.223833] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7350.296959] Key type lgssc unregistered [ 7350.406380] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.24@tcp [ 7350.431444] LNet: 389249:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7350.434755] LNetError: 389249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7350.438355] LNet: Removed LNI 192.168.203.124@tcp [ 7351.461323] LNet: Removed LNI 192.168.203.124@tcp1 [ 7351.795845] Key type .llcrypt unregistered [ 7351.797229] Key type ._llcrypt unregistered [ 7358.486863] Key type ._llcrypt registered [ 7358.487974] Key type .llcrypt registered [ 7358.531825] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7358.656392] Key type .llcrypt unregistered [ 7358.658250] Key type ._llcrypt unregistered [ 7360.753902] Key type ._llcrypt registered [ 7360.755417] Key type .llcrypt registered [ 7360.797853] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7360.902620] Key type .llcrypt unregistered [ 7360.904364] Key type ._llcrypt unregistered [ 7365.458948] Key type ._llcrypt registered [ 7365.460022] Key type .llcrypt registered [ 7365.500265] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7365.612271] Key type .llcrypt unregistered [ 7365.613812] Key type ._llcrypt unregistered [ 7371.707328] Key type ._llcrypt registered [ 7371.708871] Key type .llcrypt registered [ 7371.748212] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7371.863466] Key type .llcrypt unregistered [ 7371.864904] Key type ._llcrypt unregistered [ 7379.149143] Key type ._llcrypt registered [ 7379.150424] Key type .llcrypt registered [ 7379.196102] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 7379.611672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7379.662248] alg: No test for adler32 (adler32-zlib) [ 7380.536746] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7380.653496] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 7382.247133] Key type lgssc registered [ 7382.662521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7390.501941] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing lnet_if_list [ 7390.966171] LNet: Added LNI 192.168.203.124@tcp1 [8/256/0/180] [ 7394.594045] loop0: detected capacity change from 0 to 204800000 [ 7394.665901] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7394.671591] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7394.677426] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7408.024583] loop0: detected capacity change from 0 to 204800000 [ 7408.097513] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7408.101820] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7408.106186] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7421.910894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7421.924171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7423.066242] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7423.086266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7423.155981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7424.876064] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7427.979758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7428.032801] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7428.048307] Lustre: 394685:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7428.159762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7429.620933] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7432.257504] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7434.378526] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7436.370419] loop0: detected capacity change from 0 to 204800000 [ 7451.494313] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7451.586069] Lustre: sv29105-OST0000: new disk, initializing [ 7451.587907] Lustre: srv-sv29105-OST0000: No data found on store. Initialize space. [ 7451.591257] Lustre: 396761:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7451.612235] Lustre: sv29105-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7453.943601] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7457.532840] Lustre: server umount sv29105-OST0000 complete [ 7462.449789] loop0: detected capacity change from 0 to 204800000 [ 7466.401897] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7466.507121] Lustre: *** cfs_fail_loc=727, val=1*** [ 7466.513252] Lustre: sv29105-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7468.747666] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7474.592142] Lustre: server umount sv29105-OST0000 complete [ 7479.263875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7479.266807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7479.272648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7481.824066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7481.824586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7485.273114] Lustre: server umount lustre-MDT0000 complete [ 7486.612280] LustreError: 395425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910408 with bad export cookie 6608774871712765854 [ 7486.615818] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7486.617880] LustreError: 395425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7486.735373] Lustre: server umount lustre-MDT0001 complete [ 7490.598129] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7491.659510] Key type lgssc unregistered [ 7491.784363] LNet: 400924:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7491.787767] LNetError: 400924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7491.789787] LNet: Removed LNI 192.168.203.124@tcp [ 7492.837332] LNet: Removed LNI 192.168.203.124@tcp1 [ 7493.163100] Key type .llcrypt unregistered [ 7493.164790] Key type ._llcrypt unregistered [ 7501.261686] Key type ._llcrypt registered [ 7501.262924] Key type .llcrypt registered [ 7501.299673] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 7501.754897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7501.794229] alg: No test for adler32 (adler32-zlib) [ 7502.661836] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7502.748072] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 7504.335155] Key type lgssc registered [ 7504.743333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7507.754765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7507.764888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7508.934287] LNetError: 401529:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.124@tcp1 has no available nets [ 7508.962277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7510.554262] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7513.738343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7513.852331] LNetError: 401529:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.124@tcp1 has no available nets [ 7513.855450] LNetError: 401529:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [ 7513.903517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7515.431914] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7518.140479] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7520.228497] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7524.900191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7524.982478] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7525.054528] LNetError: 401529:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.124@tcp1 has no available nets [ 7525.058210] LNetError: 401529:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [ 7525.077066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7527.299159] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7530.136068] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7533.546398] LNetError: 401529:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.124@tcp1 has no available nets [ 7533.551418] LNetError: 401529:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [ 7542.503236] Lustre: server umount lustre-MDT0000 complete [ 7542.707619] LustreError: 404722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7542.717249] LustreError: 404722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7543.777175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7543.778500] LustreError: 402216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7543.782427] Lustre: Skipped 2 previous similar messages [ 7543.789214] LustreError: 402216:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7543.884422] LustreError: 402201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910466 with bad export cookie 15198733481431645576 [ 7543.887583] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7543.890358] LustreError: 402201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7547.818047] LustreError: 402214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7547.826031] LustreError: 402214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7547.826948] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.24@tcp (stopping) [ 7548.896282] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7548.902598] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7550.064511] Lustre: server umount lustre-MDT0001 complete [ 7552.461446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7555.585286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7569.376193] LustreError: 401530:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000009869cc4 x1875808536783488/t0(0) o250->MGC192.168.203.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7569.509916] LNetError: 401529:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.124@tcp1 has no available nets [ 7569.513192] LNetError: 401529:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 5 previous similar messages [ 7569.520431] LustreError: 404044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7569.526330] LustreError: 404044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7569.546876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7569.569440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 7570.941887] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7574.006192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7574.133106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7575.142201] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7575.651577] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7578.375663] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7578.961838] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7580.862659] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7588.508223] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 7 sec [ 7589.480264] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7589.855825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7589.859085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.863507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7590.367923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7594.978488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7594.981775] Lustre: Skipped 1 previous similar message [ 7600.100417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7600.102687] Lustre: Skipped 1 previous similar message [ 7604.191126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7604.250645] Lustre: server umount lustre-OST0000 complete [ 7609.824219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7609.827290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7609.832027] Lustre: Skipped 1 previous similar message [ 7609.834350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7609.836655] Lustre: Skipped 1 previous similar message [ 7611.669234] Lustre: server umount lustre-MDT0000 complete [ 7612.952559] LustreError: 402201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910535 with bad export cookie 15198733481431652821 [ 7612.955688] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7612.957686] LustreError: 402201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7613.083353] Lustre: server umount lustre-MDT0001 complete [ 7616.929122] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7618.160418] Key type lgssc unregistered [ 7618.308376] LNet: 408490:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7618.310896] LNetError: 408490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7618.319289] LNet: Removed LNI 192.168.203.124@tcp [ 7618.681783] Key type .llcrypt unregistered [ 7618.682989] Key type ._llcrypt unregistered [ 7628.358269] Key type ._llcrypt registered [ 7628.359407] Key type .llcrypt registered [ 7628.402643] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 7628.719446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7628.727638] alg: No test for adler32 (adler32-zlib) [ 7629.584822] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7629.675233] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 7631.263165] Key type lgssc registered [ 7631.641669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7634.226290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7634.233478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7635.357270] LNetError: 409194:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.124@tcp1 has no available nets [ 7635.378652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7636.885290] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7639.986094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7640.068062] LNetError: 409194:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.124@tcp1 has no available nets [ 7640.071969] LNetError: 409194:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 4 previous similar messages [ 7640.112401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7641.577686] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7644.095365] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7645.983962] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7650.080863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7650.085478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7650.087724] Lustre: Skipped 1 previous similar message [ 7653.143089] Lustre: server umount lustre-MDT0000 complete [ 7654.417738] LustreError: 410618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910576 with bad export cookie 13667513894459132623 [ 7654.420427] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7654.423488] LustreError: 410618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7654.535496] Lustre: server umount lustre-MDT0001 complete [ 7659.600263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.644934] LNetError: 409194:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.124@tcp1 has no available nets [ 7659.648632] LNetError: 409194:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 5 previous similar messages [ 7659.697647] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7659.707303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7659.783402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7661.124981] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7664.058244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7664.100865] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7664.111681] Lustre: 412468:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7664.179489] LNetError: 409194:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.124@tcp1 has no available nets [ 7664.184222] LNetError: 409194:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 1 previous similar message [ 7664.220877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7665.652551] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7668.103413] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7669.949900] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7674.786959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7674.879205] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7674.883787] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7674.887739] LustreError: 414350:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -2 [ 7674.890968] LustreError: 414350:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-OST0000 [ 7674.894377] LustreError: 414350:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7674.925654] Lustre: server umount lustre-OST0000 complete [ 7674.928126] LustreError: 414350:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7680.226523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7680.311390] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7680.382038] LNetError: 409194:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.124@tcp1 has no available nets [ 7680.386259] LNetError: 409194:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [ 7680.402138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7682.678326] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7685.647385] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7687.203346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 7688.943969] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7688.979303] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7689.027958] Lustre: lustre-OST0001: new disk, initializing [ 7689.030569] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7689.034164] Lustre: 415891:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7689.056096] LNetError: 409194:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.124@tcp1 has no available nets [ 7689.059826] LNetError: 409194:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 7 previous similar messages [ 7689.079169] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7691.484461] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7694.624297] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7695.843557] Lustre: server umount lustre-OST0001 complete [ 7696.897765] LustreError: 414817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7696.905232] LustreError: 414817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7697.377839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7697.377934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7697.381314] LustreError: Skipped 1 previous similar message [ 7697.383029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7697.390425] Lustre: Skipped 2 previous similar messages [ 7701.985275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7701.986282] LustreError: 414817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7701.988550] Lustre: Skipped 2 previous similar messages [ 7701.995798] LustreError: 414817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7703.446099] Lustre: server umount lustre-OST0000 complete [ 7705.055859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7705.059063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7705.063952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7707.105426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7712.224144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7712.226135] Lustre: Skipped 3 previous similar messages [ 7719.391198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7719.484806] Lustre: server umount lustre-MDT0000 complete [ 7721.048285] LustreError: 412459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910643 with bad export cookie 13667513894459133162 [ 7721.051122] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7721.055040] LustreError: 412459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7721.184714] Lustre: server umount lustre-MDT0001 complete [ 7725.097081] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7726.216196] Key type lgssc unregistered [ 7726.344334] LNet: 417846:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7726.346589] LNetError: 417846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7726.352292] LNet: Removed LNI 192.168.203.124@tcp [ 7726.662335] Key type .llcrypt unregistered [ 7726.663743] Key type ._llcrypt unregistered [ 7758.704646] Key type ._llcrypt registered [ 7758.706379] Key type .llcrypt registered [ 7758.749665] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 7765.720330] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 7766.103522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7766.152513] alg: No test for adler32 (adler32-zlib) [ 7767.025861] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 7767.130127] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 7768.727214] Key type lgssc registered [ 7769.146926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7782.437972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7782.446086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7783.543421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7783.556299] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7783.596384] Lustre: lustre-MDT0000: new disk, initializing [ 7783.630651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7783.639422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7785.322611] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7789.974208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7790.022064] Lustre: 422750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7790.047188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7790.049799] Lustre: Skipped 1 previous similar message [ 7790.087649] Lustre: lustre-MDT0001: new disk, initializing [ 7790.123923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7790.136483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7790.141035] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7791.668381] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7795.710469] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7798.464305] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7802.255580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7802.560696] Lustre: lustre-OST0000: new disk, initializing [ 7802.567345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7802.573160] Lustre: 424718:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7802.648984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7808.466811] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7811.649743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7811.666028] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7811.789784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7816.298189] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7819.644355] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7819.803057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7822.863779] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7823.011078] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7827.434271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7827.436443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7827.457882] LustreError: Skipped 1 previous similar message [ 7827.459786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7827.496713] Lustre: Skipped 1 previous similar message [ 7830.827733] Lustre: server umount lustre-OST0000 complete [ 7836.128296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7836.134257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7836.150981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7836.153348] Lustre: Skipped 1 previous similar message [ 7837.669143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7837.671535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7837.691029] Lustre: Skipped 1 previous similar message [ 7840.667733] Lustre: server umount lustre-MDT0000 complete [ 7843.462208] LustreError: 422743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910765 with bad export cookie 12082321710565309502 [ 7843.465274] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7843.473189] LustreError: 422743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7843.692722] Lustre: server umount lustre-MDT0001 complete [ 7853.039460] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:39:33 (1788910773) [ 7862.000435] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:39:42 (1788910782) [ 7870.372061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7870.642550] LustreError: 427469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7870.651843] LustreError: 427469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7870.682597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7874.752161] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7876.065126] LustreError: 427470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7880.161661] LustreError: 427469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7882.420039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7882.680491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7886.337562] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7891.891758] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7895.923550] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7900.403205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7900.644080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7903.868808] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7908.582770] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7921.957886] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7923.244732] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7924.513810] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7925.992526] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7927.646643] Lustre: Failing over lustre-MDT0000 [ 7927.888034] Lustre: server umount lustre-MDT0000 complete [ 7928.800075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7928.805646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7928.817602] LustreError: 428198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7935.458800] LustreError: 427470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7935.475056] LustreError: 427470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7944.091271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7944.186385] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7944.391014] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7944.430389] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7947.563133] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7947.689917] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7949.799205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7949.836367] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 7949.856547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7954.394076] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7955.385364] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7958.304236] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7959.707803] Lustre: Failing over lustre-MDT0000 [ 7959.940466] Lustre: server umount lustre-MDT0000 complete [ 7960.035471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7960.038972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7960.041568] Lustre: Skipped 3 previous similar messages [ 7960.043035] LustreError: 429957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7960.043045] LustreError: 429957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7974.791962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7974.890279] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7975.097230] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7977.439983] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7978.414815] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7980.519178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7980.522926] Lustre: Skipped 2 previous similar messages [ 7980.544471] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 7980.574313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7983.526766] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7984.486500] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7990.753471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7990.754125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7990.754815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7990.760809] Lustre: Skipped 2 previous similar messages [ 7994.378969] Lustre: server umount lustre-MDT0000 complete [ 7995.876829] LustreError: 427470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7995.895101] LustreError: 427470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 7996.550395] LustreError: 427449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788910918 with bad export cookie 12082321710565312197 [ 7996.554846] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7996.562211] LustreError: 427449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7996.764230] Lustre: server umount lustre-MDT0001 complete [ 8009.568950] Lustre: server umount lustre-OST0000 complete [ 8019.378907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8019.687403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8019.691848] Lustre: Skipped 1 previous similar message [ 8022.504480] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8028.413471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8031.355977] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8036.411765] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8040.292252] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8045.085359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8048.756288] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8053.081512] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8064.747351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8074.224030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8074.230961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8074.243529] Lustre: Skipped 1 previous similar message [ 8074.246285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8074.249157] Lustre: Skipped 2 previous similar messages [ 8077.881460] Lustre: server umount lustre-OST0000 complete [ 8085.475681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8085.479751] Lustre: Skipped 5 previous similar messages [ 8086.412947] Lustre: server umount lustre-MDT0000 complete [ 8088.351567] LustreError: 433802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911010 with bad export cookie 12082321710565312918 [ 8088.355103] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8088.358454] LustreError: 433802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8088.503260] Lustre: server umount lustre-MDT0001 complete [ 8093.495559] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 8094.825992] Key type lgssc unregistered [ 8094.977466] LNet: 437373:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8094.981382] LNetError: 437373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8094.991378] LNet: Removed LNI 192.168.203.124@tcp [ 8095.394156] Key type .llcrypt unregistered [ 8095.395709] Key type ._llcrypt unregistered [ 8107.325804] Key type ._llcrypt registered [ 8107.327779] Key type .llcrypt registered [ 8107.389057] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 8107.821311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8107.875558] alg: No test for adler32 (adler32-zlib) [ 8108.816360] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8108.956886] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 8110.559173] Key type lgssc registered [ 8111.188301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8114.974616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8114.988527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8116.189311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8117.992822] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8121.765924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8121.940109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8123.699091] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8126.793355] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8129.057667] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8132.244775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8132.408772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8135.124915] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8138.497481] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8141.611846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8157.663986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8157.667607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8157.674213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8158.178250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8158.179109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8158.183045] Lustre: Skipped 1 previous similar message [ 8158.194600] Lustre: Skipped 1 previous similar message [ 8160.312260] Lustre: server umount lustre-MDT0000 complete [ 8161.792389] LustreError: 438848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911084 with bad export cookie 12689351366855125195 [ 8161.796847] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8161.797441] LustreError: 438848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8161.929717] Lustre: server umount lustre-MDT0001 complete [ 8172.830612] Lustre: server umount lustre-OST0000 complete [ 8178.466686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8178.673941] LustreError: 442594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8178.682327] LustreError: 442594:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8178.716876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8180.642354] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8183.778730] LustreError: 442595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8184.512913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8184.665791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8186.295973] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8189.228385] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8191.484825] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8194.316656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8194.504321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8196.992676] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8200.114736] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8204.777443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 8213.985601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8213.987508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8213.997062] Lustre: Skipped 2 previous similar messages [ 8214.001469] Lustre: Skipped 2 previous similar messages [ 8219.106164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8219.109244] Lustre: Skipped 3 previous similar messages [ 8219.311964] Lustre: server umount lustre-MDT0000 complete [ 8221.059112] LustreError: 442576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911143 with bad export cookie 12689351366855126294 [ 8221.062639] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8221.066510] LustreError: 442576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8221.206867] Lustre: server umount lustre-MDT0001 complete [ 8233.332427] Lustre: server umount lustre-OST0000 complete [ 8240.115643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8240.351543] LustreError: 446500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8240.400556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8242.325972] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8245.728248] LustreError: 446501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8246.015733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8248.097462] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8251.156093] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8253.755794] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8256.560669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8259.482890] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8262.738328] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8264.934922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 8270.305821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8270.309606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8270.316552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8271.771185] Lustre: server umount lustre-OST0000 complete [ 8275.354323] Lustre: server umount lustre-MDT0000 complete [ 8276.448546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8276.449466] LustreError: 446496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8276.454195] Lustre: Skipped 1 previous similar message [ 8276.744153] LustreError: 447234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911198 with bad export cookie 12689351366855127316 [ 8276.746704] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8276.748860] LustreError: 447234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8282.931949] Lustre: server umount lustre-MDT0001 complete [ 8286.750637] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 8287.814198] Key type lgssc unregistered [ 8287.946347] LNet: 450028:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8287.949119] LNetError: 450028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8287.958281] LNet: Removed LNI 192.168.203.124@tcp [ 8288.267074] Key type .llcrypt unregistered [ 8288.268348] Key type ._llcrypt unregistered [ 8297.981577] Key type ._llcrypt registered [ 8297.982750] Key type .llcrypt registered [ 8298.021660] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 8298.353595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8298.380086] alg: No test for adler32 (adler32-zlib) [ 8299.234974] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8299.331528] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 8300.911164] Key type lgssc registered [ 8301.297222] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8304.023503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8304.030924] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8304.034101] LustreError: 451444:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8304.042856] LustreError: 451444:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8304.045355] LustreError: 451444:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8304.048225] LustreError: 451444:0:(tgt_mount.c:2671:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8304.050990] LustreError: 451444:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8306.579259] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8306.581929] LustreError: 451840:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8306.589983] LustreError: 451840:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8306.592278] LustreError: 451840:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8306.594426] LustreError: 451840:0:(tgt_mount.c:2671:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8306.597149] LustreError: 451840:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8312.827491] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 8313.870226] Key type lgssc unregistered [ 8314.008479] LNet: 452841:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8314.011889] LNetError: 452841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8314.024217] LNet: Removed LNI 192.168.203.124@tcp [ 8314.325465] Key type .llcrypt unregistered [ 8314.327166] Key type ._llcrypt unregistered [ 8320.230366] Key type ._llcrypt registered [ 8320.231702] Key type .llcrypt registered [ 8320.268453] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 8326.527618] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 8326.838291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8326.894189] alg: No test for adler32 (adler32-zlib) [ 8327.750238] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8327.828870] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 8329.415250] Key type lgssc registered [ 8329.804722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8340.969120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8340.973501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8342.048049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8342.057347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8342.089717] Lustre: lustre-MDT0000: new disk, initializing [ 8342.109295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8342.115427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8343.388437] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8347.377276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8347.415930] Lustre: 455970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8347.436481] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8347.438166] Lustre: Skipped 1 previous similar message [ 8347.470130] Lustre: lustre-MDT0001: new disk, initializing [ 8347.498393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8347.508755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8347.513513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8348.778774] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8352.259734] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8353.865114] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8356.088983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8356.194445] Lustre: lustre-OST0000: new disk, initializing [ 8356.197461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8356.201818] Lustre: 457937:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8356.232065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8358.118190] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8361.769152] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8361.974953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8361.978413] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8362.009066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8363.200227] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8363.278367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8364.675159] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8364.748442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8367.074138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8367.074770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8367.076817] LustreError: Skipped 1 previous similar message [ 8367.081739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8367.084642] Lustre: Skipped 1 previous similar message [ 8371.471518] Lustre: server umount lustre-OST0000 complete [ 8373.216089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8373.219226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8373.224490] Lustre: Skipped 1 previous similar message [ 8373.226843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8377.312184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8377.312545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8378.835569] Lustre: server umount lustre-MDT0000 complete [ 8380.068196] LustreError: 455960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911302 with bad export cookie 7209293552702608500 [ 8380.071737] LustreError: 455960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8380.072908] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8380.192649] Lustre: server umount lustre-MDT0001 complete [ 8382.915114] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8383.457028] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 19:48:24 (1788911304) [ 8385.592942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8385.767198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8387.053618] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8389.777088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8389.890186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8391.147252] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8393.387324] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8395.076767] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8397.165741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8399.123729] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8401.412809] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8403.557371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8403.612635] Lustre: lustre-OST0001: new disk, initializing [ 8403.614240] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8403.616532] Lustre: 463227:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8403.645682] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8403.647423] Lustre: Skipped 1 previous similar message [ 8405.482292] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8409.020792] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8410.108863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8410.111827] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8410.137768] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8419.762268] Lustre: lustre-OST0001: Not available for connect from 192.168.203.24@tcp (stopping) [ 8419.763962] Lustre: Skipped 1 previous similar message [ 8420.324924] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8420.326923] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8421.368615] Lustre: server umount lustre-OST0001 complete [ 8425.448205] LustreError: 463938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8425.453150] LustreError: 463938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8430.565117] LustreError: 463949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8430.572174] LustreError: 463949:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8435.682332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8435.683169] LustreError: 462214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8435.684173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8435.684178] Lustre: Skipped 2 previous similar messages [ 8435.687537] Lustre: Skipped 2 previous similar messages [ 8435.694703] LustreError: 462214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8435.698214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8435.705710] LustreError: Skipped 1 previous similar message [ 8439.438977] Lustre: server umount lustre-OST0000 complete [ 8445.920095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8445.922196] Lustre: Skipped 4 previous similar messages [ 8446.931840] Lustre: server umount lustre-MDT0000 complete [ 8448.210314] LustreError: 460375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911370 with bad export cookie 7209293552702609368 [ 8448.213524] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8448.213571] LustreError: 460375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8454.379514] Lustre: server umount lustre-MDT0001 complete [ 8457.920260] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 8459.129645] Key type lgssc unregistered [ 8459.259633] LNet: 465213:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8459.263322] LNetError: 465213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8459.273403] LNet: Removed LNI 192.168.203.124@tcp [ 8459.581329] Key type .llcrypt unregistered [ 8459.583186] Key type ._llcrypt unregistered [ 8477.805605] Key type ._llcrypt registered [ 8477.806818] Key type .llcrypt registered [ 8477.844613] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 8478.248340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8478.258964] alg: No test for adler32 (adler32-zlib) [ 8479.106208] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8479.196439] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 8480.775132] Key type lgssc registered [ 8481.129304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8483.721863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8483.730828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8484.841112] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8484.848510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8484.910949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8486.305598] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8489.252024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8489.289217] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8489.300459] Lustre: 467985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8489.394945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8490.806080] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8493.376073] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8495.250728] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8496.096478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8496.097164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8496.101191] Lustre: Skipped 1 previous similar message [ 8501.216438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8501.218901] Lustre: Skipped 2 previous similar messages [ 8502.038207] Lustre: server umount lustre-MDT0000 complete [ 8503.394383] LustreError: 467977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911425 with bad export cookie 13614129079928450848 [ 8503.397163] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8503.400081] LustreError: 467977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8503.515895] Lustre: server umount lustre-MDT0001 complete [ 8506.699869] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:50:27 (1788911427) [ 8508.869709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8509.023212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8510.352245] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8513.103468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8513.229776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8514.540291] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8516.765162] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8518.512278] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8520.607809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8520.667231] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8520.722193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8522.457811] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8524.735735] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8528.337748] Lustre: Permanently deactivating lustre-OST0000 [ 8528.342047] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8532.973973] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8534.068338] LustreError: 473474:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8534.071357] LustreError: 473474:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 8534.073716] LustreError: 473474:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8534.650855] LustreError: 473522:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 8534.653947] LustreError: 473522:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8541.324915] Lustre: server umount lustre-OST0000 complete [ 8542.212848] LustreError: 473772:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 8542.216242] LustreError: 473772:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8543.712579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8543.713250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8543.717428] Lustre: Skipped 1 previous similar message [ 8548.832246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8548.835234] Lustre: Skipped 2 previous similar messages [ 8552.955900] Lustre: server umount lustre-MDT0000 complete [ 8553.952987] LustreError: 470493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8553.958631] LustreError: 470493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8554.212404] LustreError: 470474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911476 with bad export cookie 13614129079928451415 [ 8554.216738] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8554.217806] LustreError: 470474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8564.752481] Lustre: server umount lustre-MDT0001 complete [ 8567.737370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8569.212818] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8570.182126] LustreError: 474929:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8572.439152] Lustre: 475267:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.124@tcp [ 8573.171917] Lustre: server umount MGS complete [ 8577.067152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8577.232528] LustreError: 475989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8577.237589] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8577.241701] Lustre: Skipped 1 previous similar message [ 8577.263520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8578.757961] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8581.707545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8581.805064] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8583.209343] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8585.734212] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8587.564431] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8588.178794] Lustre: Permanently reactivating lustre-OST0000 [ 8588.184130] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8588.186089] Lustre: Skipped 2 previous similar messages [ 8613.856048] LustreError: 477774:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8616.166062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8616.269636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8616.272467] Lustre: Skipped 1 previous similar message [ 8618.116903] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8620.439090] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8633.823762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8633.826090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8633.829866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8634.849508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8638.537594] Lustre: server umount lustre-OST0000 complete [ 8639.968148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8639.972631] Lustre: Skipped 2 previous similar messages [ 8645.088334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8645.090480] Lustre: Skipped 5 previous similar messages [ 8654.303177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8654.365441] Lustre: server umount lustre-MDT0000 complete [ 8655.328273] LustreError: 477422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8655.334629] LustreError: 477422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8655.703685] LustreError: 476724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911577 with bad export cookie 13614129079928452451 [ 8655.708346] LustreError: 476724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8655.708742] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8655.817966] Lustre: server umount lustre-MDT0001 complete [ 8659.630336] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 8660.697469] Key type lgssc unregistered [ 8660.827323] LNet: 480665:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8660.830847] LNetError: 480665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8660.838316] LNet: Removed LNI 192.168.203.124@tcp [ 8661.171919] Key type .llcrypt unregistered [ 8661.173075] Key type ._llcrypt unregistered [ 8667.267149] Key type ._llcrypt registered [ 8667.268952] Key type .llcrypt registered [ 8667.314903] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 8673.622544] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 8674.018592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8674.028111] alg: No test for adler32 (adler32-zlib) [ 8674.886236] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8674.981541] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 8676.567176] Key type lgssc registered [ 8676.899721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8687.912596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8687.917258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8688.995343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8689.004515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8689.044233] Lustre: lustre-MDT0000: new disk, initializing [ 8689.063452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8689.068888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8690.355822] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8694.316310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8694.355104] Lustre: 483793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8694.376678] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8694.378805] Lustre: Skipped 1 previous similar message [ 8694.411734] Lustre: lustre-MDT0001: new disk, initializing [ 8694.441543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8694.451892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8694.454727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8695.798670] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8699.448891] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8701.176950] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8703.418582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8703.523269] Lustre: lustre-OST0000: new disk, initializing [ 8703.525563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8703.529380] Lustre: 485760:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8703.560233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8705.445498] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8709.118254] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8710.512521] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8713.210605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8713.215638] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8713.244593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8713.637371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 8714.934643] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8715.009097] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8718.307220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8718.307364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8718.310146] LustreError: Skipped 1 previous similar message [ 8718.315369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8718.318292] Lustre: Skipped 1 previous similar message [ 8721.743555] Lustre: server umount lustre-OST0000 complete [ 8723.424626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8723.425402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8723.429476] Lustre: Skipped 2 previous similar messages [ 8723.433794] Lustre: Skipped 1 previous similar message [ 8728.544353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8728.546687] Lustre: Skipped 1 previous similar message [ 8733.664238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8733.667341] Lustre: Skipped 1 previous similar message [ 8737.247071] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8737.309154] Lustre: server umount lustre-MDT0000 complete [ 8738.485323] LustreError: 483783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911660 with bad export cookie 8662544623429003261 [ 8738.487819] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8738.488824] LustreError: 483783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8738.596819] Lustre: server umount lustre-MDT0001 complete [ 8741.285654] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8741.854061] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 19:54:23 (1788911663) [ 8744.024888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8744.159202] LustreError: 488243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8744.165775] LustreError: 488243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8744.181714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8745.508848] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8748.260885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8748.385639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8749.683751] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8751.978740] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8753.781378] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8755.990300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8757.921846] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8760.263508] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8760.859203] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8771.576686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8773.599880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8773.602367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8773.607802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8774.625945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8779.147433] Lustre: server umount lustre-OST0000 complete [ 8784.351876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8784.355221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8784.360655] Lustre: Skipped 1 previous similar message [ 8784.363028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8784.365068] Lustre: Skipped 2 previous similar messages [ 8786.448517] Lustre: server umount lustre-MDT0000 complete [ 8787.548817] LustreError: 488269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911709 with bad export cookie 8662544623429004136 [ 8787.550734] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8787.552182] LustreError: 488269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8787.653376] Lustre: server umount lustre-MDT0001 complete [ 8790.891690] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 8791.845864] Key type lgssc unregistered [ 8791.972297] LNet: 491988:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8791.974859] LNetError: 491988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8791.981266] LNet: Removed LNI 192.168.203.124@tcp [ 8792.271460] Key type .llcrypt unregistered [ 8792.272853] Key type ._llcrypt unregistered [ 8799.520728] Key type ._llcrypt registered [ 8799.521654] Key type .llcrypt registered [ 8799.551975] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 8799.641191] Key type .llcrypt unregistered [ 8799.642653] Key type ._llcrypt unregistered [ 8806.419975] Key type ._llcrypt registered [ 8806.421675] Key type .llcrypt registered [ 8806.458566] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 8806.862634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8806.868703] alg: No test for adler32 (adler32-zlib) [ 8807.720313] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8807.802700] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 8809.383126] Key type lgssc registered [ 8809.716000] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8812.151180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8812.156695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8813.284966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8814.489810] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8817.067889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8817.192178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8818.991229] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8821.099768] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8822.240447] LustreError: 494174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8822.245702] LustreError: 494174:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8823.169438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8823.261420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8823.302743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8824.512427] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8833.507597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8833.507646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8833.511515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8833.513833] LustreError: Skipped 1 previous similar message [ 8837.581084] Lustre: server umount lustre-OST0000 complete [ 8841.697337] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8841.700483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8841.705306] Lustre: Skipped 1 previous similar message [ 8841.707601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8841.710686] Lustre: Skipped 1 previous similar message [ 8843.744171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8843.744601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8844.887647] Lustre: server umount lustre-MDT0000 complete [ 8846.017070] LustreError: 494155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911768 with bad export cookie 4114912670395637541 [ 8846.020022] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8846.020813] LustreError: 494155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8846.125564] Lustre: server umount lustre-MDT0001 complete [ 8849.515265] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 8850.412685] Key type lgssc unregistered [ 8850.534388] LNet: 497317:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8850.536942] LNetError: 497317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8850.547318] LNet: Removed LNI 192.168.203.124@tcp [ 8850.845230] Key type .llcrypt unregistered [ 8850.846367] Key type ._llcrypt unregistered [ 8859.385175] Key type ._llcrypt registered [ 8859.386658] Key type .llcrypt registered [ 8859.423077] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 8859.724894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8859.731863] alg: No test for adler32 (adler32-zlib) [ 8860.576730] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8860.664806] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 8862.239154] Key type lgssc registered [ 8862.573323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8865.013906] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8865.020417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8880.479439] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8885.599278] LustreError: 498683:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.124@tcp: failed processing log, type 4: rc = -110 [ 8912.223215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8914.205273] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8916.649196] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8918.768184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8918.914898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8918.935260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8920.167675] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8922.834977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8922.956185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8924.218301] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8926.479693] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8928.140653] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8932.322763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8932.323330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8932.328202] Lustre: Skipped 1 previous similar message [ 8936.462597] Lustre: server umount lustre-OST0000 complete [ 8938.463806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8938.466717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8938.471596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8938.474656] Lustre: Skipped 1 previous similar message [ 8942.559932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8942.560447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8943.763389] Lustre: server umount lustre-MDT0000 complete [ 8944.917204] LustreError: 498691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911867 with bad export cookie 8032280528903027861 [ 8944.920268] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8944.922354] LustreError: 498691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8945.030461] Lustre: server umount lustre-MDT0001 complete [ 8948.468933] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 8949.460300] Key type lgssc unregistered [ 8949.588320] LNet: 502252:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8949.590656] LNetError: 502252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8949.599325] LNet: Removed LNI 192.168.203.124@tcp [ 8949.909411] Key type .llcrypt unregistered [ 8949.911266] Key type ._llcrypt unregistered [ 8958.736441] Key type ._llcrypt registered [ 8958.738235] Key type .llcrypt registered [ 8958.782001] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 8959.224604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8959.233530] alg: No test for adler32 (adler32-zlib) [ 8960.091441] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 8960.181299] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 8961.767144] Key type lgssc registered [ 8962.170416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8964.910615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8964.918858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8966.081585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8967.511029] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8970.274092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8970.412727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8971.679861] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8973.929912] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8975.679828] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8977.794183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8977.902438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8979.787452] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 8982.240652] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8992.234075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8996.319886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8996.322546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8996.327803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8997.346530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8997.347126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8999.768952] Lustre: server umount lustre-MDT0000 complete [ 9000.538723] LustreError: 503624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.24@tcp arrived at 1788911922 with bad export cookie 5467302521097233684 [ 9000.544643] LustreError: 503624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9001.203318] LustreError: 503625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788911923 with bad export cookie 5467302521097233033 [ 9001.206367] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9001.209392] LustreError: 503625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9002.463931] LustreError: 503643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9002.464190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9002.469280] LustreError: 503643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9002.472643] Lustre: Skipped 1 previous similar message [ 9002.473263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9002.479368] Lustre: Skipped 3 previous similar messages [ 9006.495896] LustreError: 504382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9006.496447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9006.503388] LustreError: 504382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9007.317856] Lustre: server umount lustre-OST0000 complete [ 9009.706126] Lustre: server umount lustre-MDT0001 complete [ 9013.437850] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 9014.543297] Key type lgssc unregistered [ 9014.665393] LNet: 507345:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9014.668590] LNetError: 507345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9014.682266] LNet: Removed LNI 192.168.203.124@tcp [ 9015.000705] Key type .llcrypt unregistered [ 9015.002272] Key type ._llcrypt unregistered [ 9024.033803] Key type ._llcrypt registered [ 9024.034989] Key type .llcrypt registered [ 9024.079894] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 9024.444991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9024.490851] alg: No test for adler32 (adler32-zlib) [ 9025.348911] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9025.440041] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 9027.015098] Key type lgssc registered [ 9027.333763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9029.626947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9029.633773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9030.768584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9032.017405] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9034.718197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9034.847544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9036.102110] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9038.363376] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9040.101232] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9042.265509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9042.371970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9044.146708] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9046.470133] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9056.743553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9060.832259] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9060.835208] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9060.840354] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9061.859462] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9061.859877] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9064.143829] Lustre: server umount lustre-MDT0001 complete [ 9066.977095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9066.977658] LustreError: 509470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9066.980616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9066.984942] LustreError: 509470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9066.994096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9066.996275] Lustre: Skipped 2 previous similar messages [ 9071.951812] Lustre: server umount lustre-OST0000 complete [ 9072.096666] LustreError: 508739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9073.283164] Lustre: server umount lustre-MDT0000 complete [ 9077.553598] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 9078.519104] Key type lgssc unregistered [ 9078.640418] LNet: 512444:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9078.643844] LNetError: 512444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9078.657296] LNet: Removed LNI 192.168.203.124@tcp [ 9078.943110] Key type .llcrypt unregistered [ 9078.944140] Key type ._llcrypt unregistered [ 9086.652775] Key type ._llcrypt registered [ 9086.653819] Key type .llcrypt registered [ 9086.694205] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 9092.551160] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 9092.860611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9092.881055] alg: No test for adler32 (adler32-zlib) [ 9093.721212] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9093.804085] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 9095.383161] Key type lgssc registered [ 9095.748125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9106.978901] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 9111.798822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9111.804534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9112.881097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9112.891487] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9112.932055] Lustre: lustre-MDT0000: new disk, initializing [ 9112.952782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9112.959798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9114.230930] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9119.774164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9119.878441] Lustre: lustre-OST0000: new disk, initializing [ 9119.879917] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9119.882150] Lustre: Skipped 1 previous similar message [ 9119.884835] Lustre: 517445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9119.908622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9121.284410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9121.288417] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9121.297880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9121.849958] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9125.543290] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9130.654088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9130.697513] Lustre: 516428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9130.725488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9130.765568] Lustre: lustre-MDT0001: new disk, initializing [ 9130.800215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9130.811514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 9130.816222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 9132.145739] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9135.736300] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9135.805083] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9137.108118] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9140.227102] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 3 sec [ 9144.800972] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9144.801573] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9144.804056] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9144.806590] Lustre: Skipped 1 previous similar message [ 9147.856893] Lustre: server umount lustre-MDT0001 complete [ 9155.042046] LustreError: 516436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9155.045981] LustreError: 516436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9160.160571] LustreError: 518784:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9160.160638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9160.169899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9160.171472] Lustre: Skipped 2 previous similar messages [ 9165.280682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9165.281238] LustreError: 516440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9166.098803] Lustre: server umount lustre-OST0000 complete [ 9173.625698] Lustre: server umount lustre-MDT0000 complete [ 9178.351976] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 9179.416858] Key type lgssc unregistered [ 9179.554397] LNet: 521225:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9179.557380] LNetError: 521225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9179.566321] LNet: Removed LNI 192.168.203.124@tcp [ 9179.910881] Key type .llcrypt unregistered [ 9179.911940] Key type ._llcrypt unregistered [ 9191.702705] Key type ._llcrypt registered [ 9191.704117] Key type .llcrypt registered [ 9191.739687] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 9191.839431] Key type .llcrypt unregistered [ 9191.840635] Key type ._llcrypt unregistered [ 9198.316036] Key type ._llcrypt registered [ 9198.316996] Key type .llcrypt registered [ 9198.351557] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 9198.673487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9198.722499] alg: No test for adler32 (adler32-zlib) [ 9199.570935] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 9199.655289] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 9201.231178] Key type lgssc registered [ 9201.601180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9210.087671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9210.094164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9211.178262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9211.190253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9211.227494] Lustre: lustre-MDT0000: new disk, initializing [ 9211.258218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9211.269750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9212.582499] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9216.571195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9216.608112] Lustre: 524574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9216.628652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9216.630297] Lustre: Skipped 1 previous similar message [ 9216.663415] Lustre: lustre-MDT0001: new disk, initializing [ 9216.693782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9216.724038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9216.728123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9217.959736] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9221.493504] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9223.196910] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9225.259857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9225.354431] Lustre: lustre-OST0000: new disk, initializing [ 9225.355786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9225.357729] Lustre: 526541:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9225.382803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9227.160632] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9228.471523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9228.474742] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9228.507776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9230.604207] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9233.889182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9233.890596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9233.893896] Lustre: Skipped 2 previous similar messages [ 9233.896625] Lustre: Skipped 2 previous similar messages [ 9238.937660] Lustre: server umount lustre-MDT0000 complete [ 9240.180555] LustreError: 524565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912162 with bad export cookie 1723187918144605113 [ 9240.184647] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9240.186552] LustreError: 524565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9240.305691] Lustre: server umount lustre-MDT0001 complete [ 9251.804486] Lustre: server umount lustre-OST0000 complete [ 9255.053239] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:02:56 (1788912176) [ 9257.518012] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9257.655332] LustreError: 528567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9257.662436] LustreError: 528567:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9257.675506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9258.956168] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9261.725441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9261.839892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9263.140398] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9265.489746] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9267.318198] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9282.658598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9282.719674] Lustre: Found index 0 for lustre-OST0000, updating log [ 9282.774600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9284.657940] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9286.917792] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9303.269687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 9312.735904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9312.739595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9312.745564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9313.760646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9313.763219] Lustre: Skipped 2 previous similar messages [ 9319.907761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9319.909830] Lustre: Skipped 1 previous similar message [ 9322.975140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9323.039071] Lustre: server umount lustre-OST0000 complete [ 9335.565055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9335.565579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9335.568934] Lustre: Skipped 2 previous similar messages [ 9341.725101] Lustre: server umount lustre-MDT0000 complete [ 9342.896149] LustreError: 528548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912265 with bad export cookie 1723187918144606492 [ 9342.898850] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9342.900041] LustreError: 528548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9343.005537] Lustre: server umount lustre-MDT0001 complete [ 9345.471407] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:04:26 (1788912266) [ 9348.127235] LNet: 1 local NIs in recovery (showing 1): 192.168.203.124@tcp [ 9927.037383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9927.265573] LustreError: 534989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9927.316307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9929.174328] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9932.768575] LustreError: 534990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9933.188975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9933.330977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9935.287895] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9938.758167] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9941.029520] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9956.451493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9956.537833] Lustre: Found index 0 for lustre-OST0000, updating log [ 9956.647501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9958.708091] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 9961.551585] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9966.049085] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9966.052482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9966.056517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9966.058546] Lustre: Skipped 3 previous similar messages [ 9968.401442] Lustre: server umount lustre-OST0000 complete [ 9970.849285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9970.850771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9970.853225] Lustre: Skipped 1 previous similar message [ 9972.896577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9972.898949] Lustre: Skipped 4 previous similar messages [ 9975.968550] LustreError: 535716:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9975.976429] LustreError: 535716:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9975.989176] Lustre: server umount lustre-MDT0000 complete [ 9977.503087] LustreError: 536886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788912899 with bad export cookie 1723187918144607563 [ 9977.506081] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9977.508206] LustreError: 536886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9977.638962] Lustre: server umount lustre-MDT0001 complete [ 9987.103248] LNet: 1 local NIs in recovery (showing 1): 192.168.203.124@tcp [ 9995.468836] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 9998.830227] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [10012.336806] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:15:33 (1788912933) [10018.857155] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [10019.906067] Key type lgssc unregistered [10020.044677] LNet: 541637:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10020.051117] LNetError: 541637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10085.605370] LNet: Removed LNI 192.168.203.124@tcp [10085.927940] Key type .llcrypt unregistered [10085.929157] Key type ._llcrypt unregistered [10093.092824] Key type ._llcrypt registered [10093.094434] Key type .llcrypt registered [10093.146585] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [10093.546163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10093.577537] alg: No test for adler32 (adler32-zlib) [10094.428219] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [10094.521506] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [10096.111110] Key type lgssc registered [10096.457922] Lustre: Echo OBD driver; http://www.lustre.org/ [10097.813981] LNet: Added LNI 192.168.203.124@tcp100 [8/256/0/180] [10098.092163] LNet: Added LNI 192.168.203.124@tcp101 [8/256/0/180] [10098.368512] LNet: Added LNI 192.168.203.124@tcp102 [8/256/0/180] [10098.644131] LNet: Added LNI 192.168.203.124@tcp103 [8/256/0/180] [10098.926996] LNet: Added LNI 192.168.203.124@tcp104 [8/256/0/180] [10099.193734] LNet: Added LNI 192.168.203.124@tcp105 [8/256/0/180] [10099.479536] LNet: Added LNI 192.168.203.124@tcp106 [8/256/0/180] [10099.760387] LNet: Added LNI 192.168.203.124@tcp107 [8/256/0/180] [10100.075768] LNet: Added LNI 192.168.203.124@tcp108 [8/256/0/180] [10100.375279] LNet: Added LNI 192.168.203.124@tcp109 [8/256/0/180] [10100.671867] LNet: Added LNI 192.168.203.124@tcp110 [8/256/0/180] [10100.979466] LNet: Added LNI 192.168.203.124@tcp111 [8/256/0/180] [10101.292489] LNet: Added LNI 192.168.203.124@tcp112 [8/256/0/180] [10101.586384] LNet: Added LNI 192.168.203.124@tcp113 [8/256/0/180] [10101.870713] LNet: Added LNI 192.168.203.124@tcp114 [8/256/0/180] [10102.173082] LNet: Added LNI 192.168.203.124@tcp115 [8/256/0/180] [10102.459624] LNet: Added LNI 192.168.203.124@tcp116 [8/256/0/180] [10102.774392] LNet: Added LNI 192.168.203.124@tcp117 [8/256/0/180] [10103.082859] LNet: Added LNI 192.168.203.124@tcp118 [8/256/0/180] [10103.383873] LNet: Added LNI 192.168.203.124@tcp119 [8/256/0/180] [10103.681051] LNet: Added LNI 192.168.203.124@tcp120 [8/256/0/180] [10103.991181] LNet: Added LNI 192.168.203.124@tcp121 [8/256/0/180] [10104.305342] LNet: Added LNI 192.168.203.124@tcp122 [8/256/0/180] [10104.648228] LNet: Added LNI 192.168.203.124@tcp123 [8/256/0/180] [10104.976994] LNet: Added LNI 192.168.203.124@tcp124 [8/256/0/180] [10105.306079] LNet: Added LNI 192.168.203.124@tcp125 [8/256/0/180] [10105.637494] LNet: Added LNI 192.168.203.124@tcp126 [8/256/0/180] [10105.967080] LNet: Added LNI 192.168.203.124@tcp127 [8/256/0/180] [10106.263221] LNet: Added LNI 192.168.203.124@tcp128 [8/256/0/180] [10106.560786] LNet: Added LNI 192.168.203.124@tcp129 [8/256/0/180] [10106.877651] LNet: Added LNI 192.168.203.124@tcp130 [8/256/0/180] [10107.181593] LNet: Added LNI 192.168.203.124@tcp131 [8/256/0/180] [10107.485261] LNet: Added LNI 192.168.203.124@tcp132 [8/256/0/180] [10107.785224] LNet: Added LNI 192.168.203.124@tcp133 [8/256/0/180] [10108.098760] LNet: Added LNI 192.168.203.124@tcp134 [8/256/0/180] [10108.397040] LNet: Added LNI 192.168.203.124@tcp135 [8/256/0/180] [10110.463721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10110.468634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10111.508612] Lustre: 544635:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.124@tcp: can't add NID 192.168.203.124@tcp130: rc = -75 [10111.536740] LNetError: 542101:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.124 [10111.540750] LNetError: Protocol error connecting to 192.168.203.124@tcp122 at host 192.168.203.124:988: is it running a compatible version of Lustre? [10111.544227] LNetError: Skipped 2 previous similar messages [10111.584694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10111.594022] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10111.626822] Lustre: lustre-MDT0000: new disk, initializing [10111.645567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10111.651442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10111.967073] LNet: 5 local NIs in recovery (showing 5): 192.168.203.124@tcp125, 192.168.203.124@tcp126, 192.168.203.124@tcp127, 192.168.203.124@tcp122, 192.168.203.124@tcp128 [10112.609313] LNetError: 542101:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.124 [10112.611887] LNetError: 542101:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [10112.613803] LNetError: Protocol error connecting to 192.168.203.124@tcp108 at host 192.168.203.124:988: is it running a compatible version of Lustre? [10112.616827] LNetError: Skipped 6 previous similar messages [10112.901713] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10114.019362] LNetError: 544662:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.124 [10114.021662] LNetError: 544662:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [10114.023816] LNetError: Protocol error connecting to 192.168.203.124@tcp125 at host 192.168.203.124:988: is it running a compatible version of Lustre? [10114.026055] LNetError: Skipped 5 previous similar messages [10116.870752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10116.923544] Lustre: 544653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10116.955565] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10116.957730] Lustre: Skipped 1 previous similar message [10117.001564] Lustre: lustre-MDT0001: new disk, initializing [10117.015440] Lustre: 545454:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.124@tcp: can't add NID 192.168.203.124@tcp130: rc = -75 [10117.020136] Lustre: 545454:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [10117.070201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10117.085108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10117.088222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10118.112473] LNetError: 544656:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.124 [10118.116994] LNetError: Protocol error connecting to 192.168.203.124@tcp125 at host 192.168.203.124:988: is it running a compatible version of Lustre? [10118.493532] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10122.890675] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10125.751829] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10126.309116] LNetError: 542099:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.124 [10126.323517] LNetError: Protocol error connecting to 192.168.203.124@tcp125 at host 192.168.203.124:988: is it running a compatible version of Lustre? [10130.144390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10130.402285] Lustre: lustre-OST0000: new disk, initializing [10130.407754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10130.415844] Lustre: 546628:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10130.481721] Lustre: 546633:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.124@tcp: can't add NID 192.168.203.124@tcp130: rc = -75 [10130.489214] Lustre: 546633:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [10130.528887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10136.025882] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10136.676674] Lustre: 547002:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.124@tcp: can't add NID 192.168.203.124@tcp130: rc = -75 [10136.686241] Lustre: 547002:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [10136.778755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10136.798197] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10136.971690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10142.695093] LNetError: 542098:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.124 [10142.713459] LNetError: Protocol error connecting to 192.168.203.124@tcp125 at host 192.168.203.124:988: is it running a compatible version of Lustre? [10147.613340] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10157.537936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10157.541231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10157.563276] LustreError: Skipped 1 previous similar message [10157.566439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10160.830177] Lustre: server umount lustre-OST0000 complete [10163.174911] LNetError: 544664:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.124 [10163.188015] LNetError: 544664:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [10163.206735] LNetError: Protocol error connecting to 192.168.203.124@tcp120 at host 192.168.203.124:988: is it running a compatible version of Lustre? [10163.223706] LNetError: Skipped 11 previous similar messages [10167.790530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10167.794068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10167.812524] Lustre: Skipped 2 previous similar messages [10167.820248] Lustre: Skipped 1 previous similar message [10170.793079] Lustre: server umount lustre-MDT0000 complete [10172.383139] LNet: 1 local NIs in recovery (showing 1): 192.168.203.124@tcp125 [10173.939026] LustreError: 544644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913096 with bad export cookie 16781080852656386757 [10173.939966] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10173.946024] LustreError: 544644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10174.158709] Lustre: server umount lustre-MDT0001 complete [10179.920797] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [10181.305154] Key type lgssc unregistered [10181.490558] LNet: 548537:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10181.495536] LNetError: 548537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10181.499766] LNet: Removed LNI 192.168.203.124@tcp [10182.559191] LNet: Removed LNI 192.168.203.124@tcp100 [10182.561405] LNetError: 548537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10182.566232] LNetError: 548537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10182.571392] LNet: Removed LNI 192.168.203.124@tcp101 [10183.583159] LNet: Removed LNI 192.168.203.124@tcp102 [10183.586149] LNetError: 548537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10183.589781] LNetError: 548537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10183.593666] LNet: Removed LNI 192.168.203.124@tcp103 [10184.607147] LNet: Removed LNI 192.168.203.124@tcp104 [10185.631640] LNet: Removed LNI 192.168.203.124@tcp105 [10185.633611] LNetError: 548537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10185.638657] LNetError: 548537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [10185.642585] LNet: Removed LNI 192.168.203.124@tcp106 [10186.655284] LNet: Removed LNI 192.168.203.124@tcp107 [10187.679433] LNet: Removed LNI 192.168.203.124@tcp108 [10188.704415] LNet: Removed LNI 192.168.203.124@tcp109 [10189.727213] LNet: Removed LNI 192.168.203.124@tcp110 [10189.729928] LNetError: 548537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10189.735018] LNetError: 548537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [10189.740573] LNet: Removed LNI 192.168.203.124@tcp111 [10190.751153] LNet: Removed LNI 192.168.203.124@tcp112 [10191.775150] LNet: Removed LNI 192.168.203.124@tcp113 [10192.799589] LNet: Removed LNI 192.168.203.124@tcp114 [10193.823354] LNet: Removed LNI 192.168.203.124@tcp115 [10194.847166] LNet: Removed LNI 192.168.203.124@tcp116 [10195.871246] LNet: Removed LNI 192.168.203.124@tcp117 [10196.895575] LNet: Removed LNI 192.168.203.124@tcp118 [10197.920281] LNet: Removed LNI 192.168.203.124@tcp119 [10197.922318] LNetError: 548537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10197.926687] LNetError: 548537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [10197.930218] LNet: Removed LNI 192.168.203.124@tcp120 [10198.943261] LNet: Removed LNI 192.168.203.124@tcp121 [10199.968331] LNet: Removed LNI 192.168.203.124@tcp122 [10200.991124] LNet: Removed LNI 192.168.203.124@tcp123 [10202.015297] LNet: Removed LNI 192.168.203.124@tcp124 [10202.017303] LNet: Removed LNI 192.168.203.124@tcp125 [10203.039185] LNet: Removed LNI 192.168.203.124@tcp126 [10204.063096] LNet: Removed LNI 192.168.203.124@tcp127 [10205.087762] LNet: Removed LNI 192.168.203.124@tcp128 [10206.111159] LNet: Removed LNI 192.168.203.124@tcp129 [10206.113315] LNet: Removed LNI 192.168.203.124@tcp130 [10206.115094] LNet: Removed LNI 192.168.203.124@tcp131 [10206.117087] LNet: Removed LNI 192.168.203.124@tcp132 [10206.118883] LNet: Removed LNI 192.168.203.124@tcp133 [10206.120674] LNet: Removed LNI 192.168.203.124@tcp134 [10206.128858] LNet: Removed LNI 192.168.203.124@tcp135 [10206.446143] Key type .llcrypt unregistered [10206.447760] Key type ._llcrypt unregistered [10216.016905] Key type ._llcrypt registered [10216.018478] Key type .llcrypt registered [10216.065237] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [10223.545950] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [10223.907722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10223.962337] alg: No test for adler32 (adler32-zlib) [10224.848737] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [10224.969711] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [10226.575147] Key type lgssc registered [10227.102758] Lustre: Echo OBD driver; http://www.lustre.org/ [10242.056216] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 20:19:23 (1788913163) [10250.109065] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [10251.368220] Key type lgssc unregistered [10251.514370] LNet: 552601:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10251.518429] LNetError: 552601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10251.528462] LNet: Removed LNI 192.168.203.124@tcp [10251.871789] Key type .llcrypt unregistered [10251.873693] Key type ._llcrypt unregistered [10259.741253] Key type ._llcrypt registered [10259.743281] Key type .llcrypt registered [10259.792494] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [10260.406932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10260.425774] alg: No test for adler32 (adler32-zlib) [10261.316356] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [10261.453527] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [10263.056499] Key type lgssc registered [10263.618030] Lustre: Echo OBD driver; http://www.lustre.org/ [10268.585867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10268.595785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10269.703099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10269.720293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10269.763691] Lustre: lustre-MDT0000: new disk, initializing [10269.802679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10269.815214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10271.536237] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10276.642737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10276.695421] Lustre: 553788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10276.728924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10276.731054] Lustre: Skipped 1 previous similar message [10276.775060] Lustre: lustre-MDT0001: new disk, initializing [10276.816505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10276.832663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10276.838189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10278.677134] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10283.575631] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10286.420219] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10289.735507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10289.874910] Lustre: lustre-OST0000: new disk, initializing [10289.877669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10289.881385] Lustre: 555758:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10289.923096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10292.306697] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10296.523094] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10296.840169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10296.845832] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10296.891302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10297.918575] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing lnet_if_list [10299.931201] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [10299.944664] LNet: Added LNI 192.168.203.124@tcp10 [8/256/0/180] [10302.175638] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [10302.191074] LNet: Added LNI 192.168.203.124@tcp11 [8/256/0/180] [10303.854588] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [10303.869101] LNet: Added LNI 192.168.203.124@tcp12 [8/256/0/180] [10305.574742] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [10305.592061] LNet: Added LNI 192.168.203.124@tcp13 [8/256/0/180] [10307.365553] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [10307.380319] LNet: Added LNI 192.168.203.124@tcp14 [8/256/0/180] [10309.199087] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [10309.214057] LNet: Added LNI 192.168.203.124@tcp15 [8/256/0/180] [10311.120148] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [10311.137666] LNet: Added LNI 192.168.203.124@tcp16 [8/256/0/180] [10313.333722] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [10313.354906] LNet: Added LNI 192.168.203.124@tcp17 [8/256/0/180] [10315.644055] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [10315.661756] LNet: Added LNI 192.168.203.124@tcp18 [8/256/0/180] [10317.880078] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [10317.895472] LNet: Added LNI 192.168.203.124@tcp19 [8/256/0/180] [10320.072738] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [10320.087796] LNet: Added LNI 192.168.203.124@tcp20 [8/256/0/180] [10322.215384] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [10322.242922] LNet: Added LNI 192.168.203.124@tcp21 [8/256/0/180] [10324.464854] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [10324.491401] LNet: Added LNI 192.168.203.124@tcp22 [8/256/0/180] [10326.697047] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [10326.713672] LNet: Added LNI 192.168.203.124@tcp23 [8/256/0/180] [10328.996473] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [10329.011316] LNet: Added LNI 192.168.203.124@tcp24 [8/256/0/180] [10331.306313] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [10331.321414] LNet: Added LNI 192.168.203.124@tcp25 [8/256/0/180] [10333.830187] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [10333.845676] LNet: Added LNI 192.168.203.124@tcp26 [8/256/0/180] [10338.070576] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [10338.097240] LNet: Added LNI 192.168.203.124@tcp27 [8/256/0/180] [10345.011599] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [10345.030451] LNet: Added LNI 192.168.203.124@tcp28 [8/256/0/180] [10350.518844] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [10350.539915] LNet: Added LNI 192.168.203.124@tcp29 [8/256/0/180] [10356.774974] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [10356.796436] LNet: Added LNI 192.168.203.124@tcp30 [8/256/0/180] [10363.179837] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [10363.202797] LNet: Added LNI 192.168.203.124@tcp31 [8/256/0/180] [10369.482585] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [10369.500953] LNet: Added LNI 192.168.203.124@tcp32 [8/256/0/180] [10376.217356] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [10376.253887] LNet: Added LNI 192.168.203.124@tcp33 [8/256/0/180] [10384.651312] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [10384.745329] LNet: Added LNI 192.168.203.124@tcp34 [8/256/0/180] [10396.422322] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [10396.498193] LNet: Added LNI 192.168.203.124@tcp35 [8/256/0/180] [10408.328853] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [10408.405512] LNet: Added LNI 192.168.203.124@tcp36 [8/256/0/180] [10417.382543] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [10417.412775] LNet: Added LNI 192.168.203.124@tcp37 [8/256/0/180] [10425.393747] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [10425.419503] LNet: Added LNI 192.168.203.124@tcp38 [8/256/0/180] [10433.330243] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [10433.358612] LNet: Added LNI 192.168.203.124@tcp39 [8/256/0/180] [10439.349938] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [10439.383619] LNet: Added LNI 192.168.203.124@tcp40 [8/256/0/180] [10446.166086] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [10446.196430] LNet: Added LNI 192.168.203.124@tcp41 [8/256/0/180] [10452.494820] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [10452.523078] LNet: Added LNI 192.168.203.124@tcp42 [8/256/0/180] [10459.180828] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [10459.211488] LNet: Added LNI 192.168.203.124@tcp43 [8/256/0/180] [10465.912722] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [10465.932603] LNet: Added LNI 192.168.203.124@tcp44 [8/256/0/180] [10472.665726] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [10472.693707] LNet: Added LNI 192.168.203.124@tcp45 [8/256/0/180] [10479.369668] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [10479.410986] LNet: Added LNI 192.168.203.124@tcp46 [8/256/0/180] [10485.954743] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [10485.987773] LNet: Added LNI 192.168.203.124@tcp47 [8/256/0/180] [10493.464556] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [10493.521655] LNet: Added LNI 192.168.203.124@tcp48 [8/256/0/180] [10500.959575] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [10500.981233] LNet: Added LNI 192.168.203.124@tcp49 [8/256/0/180] [10508.472650] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [10508.499779] LNet: Added LNI 192.168.203.124@tcp50 [8/256/0/180] [10522.031316] Lustre: server umount lustre-MDT0000 complete [10522.592721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10522.608843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10524.960179] LustreError: 553782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913447 with bad export cookie 15836252768969071772 [10524.972666] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10525.153633] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10525.288680] Lustre: server umount lustre-MDT0001 complete [10540.750767] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [10547.612606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10550.954377] LustreError: 555749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10550.969262] LustreError: 555749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10551.000165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10554.448500] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10556.396590] LustreError: 555750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10561.128390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10561.444696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10564.526853] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10564.725935] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10570.574065] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10576.074332] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10595.301973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10595.309907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10595.320421] Lustre: Skipped 2 previous similar messages [10595.330464] Lustre: Skipped 1 previous similar message [10600.227697] Lustre: server umount lustre-OST0000 complete [10605.538345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10605.539617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10605.548283] Lustre: Skipped 1 previous similar message [10605.560202] Lustre: Skipped 1 previous similar message [10609.798491] Lustre: server umount lustre-MDT0000 complete [10610.660120] LustreError: 579981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10610.678137] LustreError: 579981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10613.090882] LustreError: 554584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913535 with bad export cookie 15836252768969074243 [10613.097462] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10613.102153] LustreError: 554584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10613.595626] Lustre: server umount lustre-MDT0001 complete [10622.891714] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [10625.118965] Key type lgssc unregistered [10625.514090] LNet: 582760:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10625.524029] LNetError: 582760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10625.530162] LNet: Removed LNI 192.168.203.124@tcp [10625.534821] LNet: Removed LNI 192.168.203.124@tcp10 [10625.545385] LNet: Removed LNI 192.168.203.124@tcp11 [10625.549646] LNet: Removed LNI 192.168.203.124@tcp12 [10625.555166] LNet: Removed LNI 192.168.203.124@tcp13 [10625.557989] LNet: Removed LNI 192.168.203.124@tcp14 [10625.561044] LNet: Removed LNI 192.168.203.124@tcp15 [10625.570909] LNet: Removed LNI 192.168.203.124@tcp16 [10625.573059] LNet: Removed LNI 192.168.203.124@tcp17 [10625.576681] LNet: Removed LNI 192.168.203.124@tcp18 [10625.580167] LNet: Removed LNI 192.168.203.124@tcp19 [10625.584525] LNet: Removed LNI 192.168.203.124@tcp20 [10625.587670] LNet: Removed LNI 192.168.203.124@tcp21 [10625.590493] LNet: Removed LNI 192.168.203.124@tcp22 [10625.593899] LNet: Removed LNI 192.168.203.124@tcp23 [10625.599497] LNet: Removed LNI 192.168.203.124@tcp24 [10625.604792] LNet: Removed LNI 192.168.203.124@tcp25 [10625.607894] LNet: Removed LNI 192.168.203.124@tcp26 [10625.612253] LNet: Removed LNI 192.168.203.124@tcp27 [10625.614450] LNet: Removed LNI 192.168.203.124@tcp28 [10625.623792] LNet: Removed LNI 192.168.203.124@tcp29 [10625.632792] LNet: Removed LNI 192.168.203.124@tcp30 [10625.636835] LNet: Removed LNI 192.168.203.124@tcp31 [10625.644665] LNet: Removed LNI 192.168.203.124@tcp32 [10625.650859] LNet: Removed LNI 192.168.203.124@tcp33 [10625.659041] LNet: Removed LNI 192.168.203.124@tcp34 [10625.673161] LNet: Removed LNI 192.168.203.124@tcp35 [10625.679462] LNet: Removed LNI 192.168.203.124@tcp36 [10625.689615] LNet: Removed LNI 192.168.203.124@tcp37 [10625.694541] LNet: Removed LNI 192.168.203.124@tcp38 [10625.701174] LNet: Removed LNI 192.168.203.124@tcp39 [10625.709090] LNet: Removed LNI 192.168.203.124@tcp40 [10625.713123] LNet: Removed LNI 192.168.203.124@tcp41 [10625.716821] LNet: Removed LNI 192.168.203.124@tcp42 [10625.719208] LNet: Removed LNI 192.168.203.124@tcp43 [10625.723747] LNet: Removed LNI 192.168.203.124@tcp44 [10625.726825] LNet: Removed LNI 192.168.203.124@tcp45 [10625.729259] LNet: Removed LNI 192.168.203.124@tcp46 [10625.731340] LNet: Removed LNI 192.168.203.124@tcp47 [10625.735648] LNet: Removed LNI 192.168.203.124@tcp48 [10625.738893] LNet: Removed LNI 192.168.203.124@tcp49 [10625.752447] LNet: Removed LNI 192.168.203.124@tcp50 [10626.376566] Key type .llcrypt unregistered [10626.378483] Key type ._llcrypt unregistered [10641.229797] Key type ._llcrypt registered [10641.231685] Key type .llcrypt registered [10641.310229] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [10653.278330] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [10653.893127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10653.909338] alg: No test for adler32 (adler32-zlib) [10654.893167] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [10655.095631] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [10656.741230] Key type lgssc registered [10657.571481] Lustre: Echo OBD driver; http://www.lustre.org/ [10682.978972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10683.011880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10684.170301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10684.200418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10684.258535] Lustre: lustre-MDT0000: new disk, initializing [10684.320113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10684.331842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10687.817841] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10695.786189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10695.844085] Lustre: 585933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10695.903296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10695.910508] Lustre: Skipped 1 previous similar message [10695.976956] Lustre: lustre-MDT0001: new disk, initializing [10696.052432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10696.080804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10696.087142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10699.097905] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10706.216714] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10710.502558] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10716.055962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10716.217544] Lustre: *** cfs_fail_loc=90f, val=0*** [10716.280299] Lustre: lustre-OST0000: new disk, initializing [10716.285754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10716.292620] Lustre: 587947:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10716.377623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10720.903122] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10724.906992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10724.917764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10724.955636] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10727.795745] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10735.077542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10735.078969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10735.090922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10735.101348] Lustre: Skipped 1 previous similar message [10740.196064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10740.200361] Lustre: Skipped 1 previous similar message [10745.323669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10747.359153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10747.484272] Lustre: server umount lustre-OST0000 complete [10750.435519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10750.437419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10750.450325] Lustre: Skipped 2 previous similar messages [10750.460234] Lustre: Skipped 2 previous similar messages [10755.553255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10755.556385] Lustre: Skipped 1 previous similar message [10756.385432] Lustre: server umount lustre-MDT0000 complete [10758.805922] LustreError: 585926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913681 with bad export cookie 2363151793795409303 [10758.819025] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10758.822707] LustreError: 585926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10758.995614] Lustre: server umount lustre-MDT0001 complete [10766.515737] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [10768.433761] Key type lgssc unregistered [10768.667315] LNet: 589853:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10768.672853] LNetError: 589853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10768.689438] LNet: Removed LNI 192.168.203.124@tcp [10769.205970] Key type .llcrypt unregistered [10769.207903] Key type ._llcrypt unregistered [10782.657692] Key type ._llcrypt registered [10782.660477] Key type .llcrypt registered [10782.737543] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [10783.472951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10783.545415] alg: No test for adler32 (adler32-zlib) [10784.532117] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [10784.715180] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [10786.367198] Key type lgssc registered [10787.257953] Lustre: Echo OBD driver; http://www.lustre.org/ [10792.311992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10792.324333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10793.631627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10796.454150] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10802.407672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10802.654175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10805.958447] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10811.210850] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10815.040983] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10819.743621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10819.972275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10824.069404] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10828.859705] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10846.579587] loop0: detected capacity change from 0 to 204800000 [10846.737324] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10846.747507] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10846.757504] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10850.849439] loop0: detected capacity change from 0 to 204800000 [10850.996607] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10851.005807] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10851.015474] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10854.634718] loop0: detected capacity change from 0 to 204800000 [10858.397036] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10858.494822] Lustre: fs15246-OST0000: new disk, initializing [10858.498252] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [10858.506392] Lustre: 594715:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10858.552317] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10862.949437] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10870.923784] loop1: detected capacity change from 0 to 204800000 [10875.011141] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10875.095288] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [10875.154910] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [10875.203277] Lustre: fs15246-MDT0000: new disk, initializing [10875.326540] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10875.347049] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10875.426296] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10876.451313] Lustre: 595837:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000009a41c7b8 x1875811978226304/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788913814 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [10878.480440] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [10885.540718] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10885.551169] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10885.565742] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10894.827604] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10894.837889] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10899.756775] Lustre: server umount fs15246-MDT0000 complete [10899.941047] LustreError: 591098:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10899.955321] LustreError: 591098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10905.060134] LustreError: 591097:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10906.021145] Lustre: server umount fs15246-OST0000 complete [10914.272320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10914.279817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10914.295725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10915.299596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10915.300100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10915.313287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10920.416919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10920.426136] Lustre: Skipped 2 previous similar messages [10925.539722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10925.545653] Lustre: Skipped 1 previous similar message [10928.096200] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10928.258240] Lustre: server umount lustre-OST0000 complete [10935.776845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10935.787070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10935.805071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10935.808067] Lustre: Skipped 1 previous similar message [10937.121370] Lustre: server umount lustre-MDT0000 complete [10939.608977] LustreError: 591084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913861 with bad export cookie 16058247510122064175 [10939.613715] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10939.615240] LustreError: 591084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10939.854689] Lustre: server umount lustre-MDT0001 complete [10946.322373] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [10948.014561] Key type lgssc unregistered [10948.205910] LNet: 598932:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10948.214050] LNetError: 598932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10948.228659] LNet: Removed LNI 192.168.203.124@tcp [10948.823283] Key type .llcrypt unregistered [10948.826780] Key type ._llcrypt unregistered [10961.086976] Key type ._llcrypt registered [10961.088568] Key type .llcrypt registered [10961.168267] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [10961.776526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10961.800931] alg: No test for adler32 (adler32-zlib) [10962.803864] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [10963.000331] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [10964.632253] Key type lgssc registered [10965.382819] Lustre: Echo OBD driver; http://www.lustre.org/ [10990.634431] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [10996.382432] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [11019.690837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11019.705832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11020.879979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11020.907416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11020.975304] Lustre: lustre-MDT0000: new disk, initializing [11021.050596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11021.063464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11024.100170] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11030.953316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11031.035565] Lustre: 603371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11031.082438] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11031.087044] Lustre: Skipped 1 previous similar message [11031.167977] Lustre: lustre-MDT0001: new disk, initializing [11031.231366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11031.262315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11031.268555] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11033.375390] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11038.543890] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11041.398628] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11044.789877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11044.933508] Lustre: lustre-OST0000: new disk, initializing [11044.936645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11044.941222] Lustre: 605338:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11044.997853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11047.679406] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [11051.043029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11051.048299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11051.104630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11052.979671] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11055.663592] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11055.786778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11058.012142] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11058.110374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11061.218523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11061.219026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11061.225904] LustreError: Skipped 1 previous similar message [11061.235364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11061.243166] Lustre: Skipped 1 previous similar message [11065.285320] Lustre: server umount lustre-OST0000 complete [11067.360442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11067.364887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11067.371581] Lustre: Skipped 1 previous similar message [11067.375804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11071.456247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11071.457129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11073.291908] Lustre: server umount lustre-MDT0000 complete [11075.584236] LustreError: 604166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788913997 with bad export cookie 10753847506487451941 [11075.593220] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11075.599831] LustreError: 604166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11075.824236] Lustre: server umount lustre-MDT0001 complete [11090.283916] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [11095.502426] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [11111.586539] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10798 sec ========= 20:33:52 (1788914032) [11112.463089] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:33:53 (1788914033) === [11114.093887] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:33:55 (1788914035) === [11134.332930] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [11135.722398] Key type lgssc unregistered [11135.925378] LNet: 611973:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11135.932531] LNetError: 611973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11135.948337] LNet: Removed LNI 192.168.203.124@tcp [11136.378356] Key type .llcrypt unregistered [11136.381355] Key type ._llcrypt unregistered