[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 473642189 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003228] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007510] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009005] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011007] pid_max: default: 32768 minimum: 301 [ 0.013054] LSM: Security Framework initializing [ 0.014068] Yama: becoming mindful. [ 0.015064] SELinux: Initializing. [ 0.016110] *** VALIDATE selinux *** [ 0.027747] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.033519] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.035110] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036135] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.037112] *** VALIDATE tmpfs *** [ 0.039086] *** VALIDATE proc *** [ 0.040325] *** VALIDATE cgroup *** [ 0.041011] *** VALIDATE cgroup2 *** [ 0.042297] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.043176] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.044009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.045033] Spectre V2 : User space: Vulnerable [ 0.046011] Speculative Store Bypass: Vulnerable [ 0.049008] debug: unmapping init [mem 0xffffffff8c859000-0xffffffff8c860fff] [ 0.051960] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.052725] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.053027] ... version: 2 [ 0.054017] ... bit width: 48 [ 0.055013] ... generic registers: 4 [ 0.056013] ... value mask: 0000ffffffffffff [ 0.057017] ... max period: 00007fffffffffff [ 0.058019] ... fixed-purpose events: 3 [ 0.059024] ... event mask: 000000070000000f [ 0.061252] rcu: Hierarchical SRCU implementation. [ 0.063427] smp: Bringing up secondary CPUs ... [ 0.064707] x86: Booting SMP configuration: [ 0.065040] .... node #0, CPUs: #1 #2 #3 [ 0.068242] smp: Brought up 1 node, 4 CPUs [ 0.070016] smpboot: Max logical packages: 1 [ 0.071022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.162591] node 0 deferred pages initialised in 88ms [ 0.166141] devtmpfs: initialized [ 0.167292] x86/mm: Memory block size: 128MB [ 0.169748] gcov: version magic: 0x41383552 [ 0.173247] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.176108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.179343] pinctrl core: initialized pinctrl subsystem [ 0.181210] [ 0.181783] ************************************************************* [ 0.184013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186012] ** ** [ 0.189016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.191011] ** ** [ 0.192010] ** This means that this kernel is built to expose internal ** [ 0.194015] ** IOMMU data structures, which may compromise security on ** [ 0.196017] ** your system. ** [ 0.198015] ** ** [ 0.201013] ** If you see this message and you are not debugging the ** [ 0.203013] ** kernel, report this immediately to your vendor! ** [ 0.206014] ** ** [ 0.208011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.210014] ************************************************************* [ 0.212975] NET: Registered protocol family 16 [ 0.215431] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.218059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.220062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.224155] cpuidle: using governor menu [ 0.225772] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.226457] PCI: Using configuration type 1 for base access [ 0.228132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.237102] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.238036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.240346] cryptd: max_cpu_qlen set to 1000 [ 0.243221] ACPI: Added _OSI(Module Device) [ 0.244010] ACPI: Added _OSI(Processor Device) [ 0.246013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.248013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.253000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.259560] ACPI: Interpreter enabled [ 0.261099] ACPI: PM: (supports S0 S3 S4 S5) [ 0.264026] ACPI: Using IOAPIC for interrupt routing [ 0.266201] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.270529] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.281882] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.284039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.286017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.288128] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.292375] acpiphp: Slot [2] registered [ 0.294141] acpiphp: Slot [5] registered [ 0.295102] acpiphp: Slot [6] registered [ 0.296149] acpiphp: Slot [7] registered [ 0.297103] acpiphp: Slot [8] registered [ 0.299112] acpiphp: Slot [9] registered [ 0.300156] acpiphp: Slot [10] registered [ 0.301132] acpiphp: Slot [3] registered [ 0.302100] acpiphp: Slot [4] registered [ 0.303086] acpiphp: Slot [11] registered [ 0.304090] acpiphp: Slot [12] registered [ 0.305102] acpiphp: Slot [13] registered [ 0.307124] acpiphp: Slot [14] registered [ 0.308079] acpiphp: Slot [15] registered [ 0.309110] acpiphp: Slot [16] registered [ 0.310079] acpiphp: Slot [17] registered [ 0.311117] acpiphp: Slot [18] registered [ 0.313124] acpiphp: Slot [19] registered [ 0.314152] acpiphp: Slot [20] registered [ 0.316109] acpiphp: Slot [21] registered [ 0.318121] acpiphp: Slot [22] registered [ 0.319151] acpiphp: Slot [23] registered [ 0.321112] acpiphp: Slot [24] registered [ 0.322113] acpiphp: Slot [25] registered [ 0.324128] acpiphp: Slot [26] registered [ 0.325132] acpiphp: Slot [27] registered [ 0.327123] acpiphp: Slot [28] registered [ 0.328136] acpiphp: Slot [29] registered [ 0.330129] acpiphp: Slot [30] registered [ 0.332089] acpiphp: Slot [31] registered [ 0.333000] PCI host bridge to bus 0000:00 [ 0.333022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.335026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.337269] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.340027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.343033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.346030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.348183] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.352096] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.355320] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.364710] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.370517] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.373022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.375018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.377026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.380719] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.383761] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.385036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387828] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.393018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.405017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.410025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.416555] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.429020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.451016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.462146] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.469023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.478022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.498025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.511563] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.517016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.524017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.543029] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.555638] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.568022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.577031] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.600025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.611744] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.617016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.625021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.646026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.656284] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.664013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.669016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.683028] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.699202] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.703511] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.706460] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.708453] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.711369] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.716239] iommu: Default domain type: Passthrough [ 0.717000] SCSI subsystem initialized [ 0.717208] ACPI: bus type USB registered [ 0.719020] usbcore: registered new interface driver usbfs [ 0.721117] usbcore: registered new interface driver hub [ 0.723206] usbcore: registered new device driver usb [ 0.726197] pps_core: LinuxPPS API ver. 1 registered [ 0.728015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.731076] PTP clock support registered [ 0.733181] EDAC MC: Ver: 3.0.0 [ 0.736320] PCI: Using ACPI for IRQ routing [ 0.738079] NetLabel: Initializing [ 0.739018] NetLabel: domain hash size = 128 [ 0.741019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.742110] NetLabel: unlabeled traffic allowed by default [ 0.745080] vgaarb: loaded [ 0.746256] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.747014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.753166] clocksource: Switched to clocksource kvm-clock [ 0.871503] VFS: Disk quotas dquot_6.6.0 [ 0.872797] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.875330] *** VALIDATE ramfs *** [ 0.876722] *** VALIDATE hugetlbfs *** [ 0.878502] pnp: PnP ACPI init [ 0.881111] pnp: PnP ACPI: found 6 devices [ 0.901412] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.905060] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.907312] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.909649] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.912437] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.915181] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.918147] NET: Registered protocol family 2 [ 0.921156] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.926847] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.931119] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.936827] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.940475] TCP: Hash tables configured (established 65536 bind 65536) [ 0.943505] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.947303] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.950805] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.954675] NET: Registered protocol family 1 [ 0.957718] RPC: Registered named UNIX socket transport module. [ 0.960529] RPC: Registered udp transport module. [ 0.962449] RPC: Registered tcp transport module. [ 0.964409] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.967180] NET: Registered protocol family 44 [ 0.969089] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.971468] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.973886] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.976531] PCI: CLS 0 bytes, default 64 [ 0.978470] Unpacking initramfs... [ 2.448204] debug: unmapping init [mem 0xffff8d59bcc54000-0xffff8d59bffbffff] [ 2.452188] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.454644] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.457641] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.119086] Initialise system trusted keyrings [ 3.120866] Key type blacklist registered [ 3.122820] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.133769] zbud: loaded [ 3.137528] *** VALIDATE nfs *** [ 3.138990] *** VALIDATE nfs4 *** [ 3.140782] pstore: using deflate compression [ 3.144232] Platform Keyring initialized [ 3.257774] NET: Registered protocol family 38 [ 3.259616] Key type asymmetric registered [ 3.261148] Asymmetric key parser 'x509' registered [ 3.262559] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.265466] io scheduler mq-deadline registered [ 3.267273] io scheduler kyber registered [ 3.269444] io scheduler bfq registered [ 3.271771] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.275507] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.278586] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.281738] ACPI: Power Button [PWRF] [ 3.287608] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.294478] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.312896] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.320551] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.345425] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.377098] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.408265] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.413332] Non-volatile memory driver v1.3 [ 3.414310] Linux agpgart interface v0.103 [ 3.446295] virtio_blk virtio1: [vda] 136464 512-byte logical blocks (69.9 MB/66.6 MiB) [ 3.449363] vda: detected capacity change from 0 to 69869568 [ 3.464362] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.469124] vdb: detected capacity change from 0 to 1073741824 [ 3.487648] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.489921] vdc: detected capacity change from 0 to 2621440000 [ 3.509204] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.512830] vdd: detected capacity change from 0 to 2621440000 [ 3.527960] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.531047] vde: detected capacity change from 0 to 4294967296 [ 3.546462] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.549795] vdf: detected capacity change from 0 to 4294967296 [ 3.556974] libphy: Fixed MDIO Bus: probed [ 3.562098] usbcore: registered new interface driver usbserial_generic [ 3.564145] usbserial: USB Serial support registered for generic [ 3.566146] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.569867] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.571811] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.574704] mousedev: PS/2 mouse device common for all mice [ 3.577863] rtc_cmos 00:05: RTC can wake from S4 [ 3.580644] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.581958] rtc_cmos 00:05: registered as rtc0 [ 3.587728] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.588190] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.591130] intel_pstate: CPU model not supported [ 3.596303] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.601152] hid: raw HID events driver (C) Jiri Kosina [ 3.603514] usbcore: registered new interface driver usbhid [ 3.605763] usbhid: USB HID core driver [ 3.607472] drop_monitor: Initializing network drop monitor service [ 3.611402] Initializing XFRM netlink socket [ 3.614398] NET: Registered protocol family 10 [ 3.617294] Segment Routing with IPv6 [ 3.618828] NET: Registered protocol family 17 [ 3.620667] mpls_gso: MPLS GSO support [ 3.625764] RAS: Correctable Errors collector initialized. [ 3.627967] AVX version of gcm_enc/dec engaged. [ 3.629744] AES CTR mode by8 optimization enabled [ 3.700541] sched_clock: Marking stable (3700490686, 0)->(4621942123, -921451437) [ 3.704295] registered taskstats version 1 [ 3.706274] Loading compiled-in X.509 certificates [ 3.708441] zswap: loaded using pool lzo/zbud [ 3.738429] Key type big_key registered [ 3.751764] Key type encrypted registered [ 3.753494] ima: No TPM chip found, activating TPM-bypass! [ 3.755548] ima: Allocated hash algorithm: sha1 [ 3.756915] ima: No architecture policies found [ 3.758110] evm: Initialising EVM extended attributes: [ 3.759267] evm: security.selinux [ 3.760303] evm: security.ima [ 3.761285] evm: security.capability [ 3.762499] evm: HMAC attrs: 0x1 [ 3.766257] rtc_cmos 00:05: setting system clock to 2026-06-12 02:03:18 UTC (1781229798) [ 3.774573] debug: unmapping init [mem 0xffffffff8d803000-0xffffffff8d9fffff] [ 3.778037] debug: unmapping init [mem 0xffffffff8c582000-0xffffffff8c858fff] [ 3.790153] Write protecting the kernel read-only data: 28672k [ 3.793801] debug: unmapping init [mem 0xffffffff8ac03000-0xffffffff8adfffff] [ 3.797015] debug: unmapping init [mem 0xffffffff8b514000-0xffffffff8b5fffff] [ 3.837186] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.845848] systemd[1]: Detected virtualization kvm. [ 3.847840] systemd[1]: Detected architecture x86-64. [ 3.849821] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.878154] systemd[1]: No hostname configured. [ 3.879886] systemd[1]: Set hostname to . [ 3.882253] random: systemd: uninitialized urandom read (16 bytes read) [ 3.884701] systemd[1]: Initializing machine ID from random generator. [ 4.017654] random: systemd: uninitialized urandom read (16 bytes read) [ 4.020744] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.026123] random: systemd: uninitialized urandom read (16 bytes read) [ 4.029405] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.034770] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.679487] device-mapper: uevent: version 1.0.3 [ 4.682194] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.457363] random: fast init done [ 5.490631] virtio_net virtio0 ens2: renamed from eth0 [ 5.523231] scsi host0: ata_piix [ 5.526338] scsi host1: ata_piix [ 5.527625] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.529494] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.879163] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.272126] random: crng init done [ 10.273294] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.724547] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.840350] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.103493] SELinux: Disabled at runtime. [ 12.163389] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.173214] systemd[1]: Detected virtualization kvm. [ 12.175260] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.671500] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.674732] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.680533] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.684582] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.687934] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.694988] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.705410] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ 12.877382] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.114673] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.436446] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.449252] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.624162] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.637980] EDAC sbridge: Ver: 1.1.2 [ 15.386504] Key type dns_resolver registered [ 15.679306] NFS: Registering the id_resolver key type [ 15.681322] Key type id_resolver registered [ 15.682967] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg355-server login: [ 42.551339] libcfs: loading out-of-tree module taints kernel. [ 42.571691] Key type ._llcrypt registered [ 42.572915] Key type .llcrypt registered [ 42.625973] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 51.724881] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 52.394652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.402035] alg: No test for adler32 (adler32-zlib) [ 53.451198] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 53.813574] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 55.439158] Key type lgssc registered [ 56.152113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 62.753303] hrtimer: interrupt took 3286523 ns [ 71.661301] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 113.602949] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 126.793048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 126.842288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 128.140830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 128.179843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 128.293871] Lustre: lustre-MDT0000: new disk, initializing [ 128.376870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 128.401858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 133.269665] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 147.813494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.942421] Lustre: 6480:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 148.010591] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 148.017176] Lustre: Skipped 1 previous similar message [ 148.147217] Lustre: lustre-MDT0001: new disk, initializing [ 148.220941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 148.258644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 148.272501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 153.110929] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.618408] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 169.112941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 169.632913] Lustre: lustre-OST0000: new disk, initializing [ 169.640672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 169.652633] Lustre: 8383:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 169.807177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.645421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 172.662531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 172.800835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 176.726515] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.903232] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.989446] Lustre: lustre-OST0001: new disk, initializing [ 191.992810] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 191.997777] Lustre: 9440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 192.071056] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 197.670551] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.209274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 199.231327] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 199.297746] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 208.073258] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 215.138534] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 221.464442] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing check_logdir /tmp/testlogs/ [ 226.544613] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing yml_node [ 232.207131] Lustre: DEBUG MARKER: Client: 2.17.53.61 [ 234.784483] Lustre: DEBUG MARKER: MDS: 2.17.53.61 [ 237.012207] Lustre: DEBUG MARKER: OSS: 2.17.53.61 [ 238.419884] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 11 22:07:11 EDT 2026 [ 253.965759] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 255.439858] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 270.817287] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 270.822471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 270.833051] Lustre: Skipped 2 previous similar messages [ 270.848957] Lustre: Skipped 3 previous similar messages [ 275.951788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 275.957796] Lustre: Skipped 3 previous similar messages [ 277.237534] Lustre: server umount lustre-MDT0000 complete [ 281.073878] LustreError: 7411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230075 with bad export cookie 13350388949191369522 [ 281.080761] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 281.084729] LustreError: 7411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 286.176594] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.178390] LustreError: 11060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 286.191115] Lustre: Skipped 1 previous similar message [ 286.192132] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 286.231687] LustreError: 11060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 290.273113] LustreError: 6487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 290.273247] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 290.301825] LustreError: 6487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 290.324793] Lustre: Skipped 2 previous similar messages [ 295.391118] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 295.392207] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 295.392279] LustreError: 12378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 295.428297] Lustre: Skipped 1 previous similar message [ 295.618878] Lustre: server umount lustre-MDT0001 complete [ 299.394560] Lustre: server umount lustre-OST0000 complete [ 302.639596] Lustre: server umount lustre-OST0001 complete [ 309.019460] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 318.247863] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 354.329423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 354.611992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 354.662725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 354.783935] Lustre: lustre-MDT0000: new disk, initializing [ 354.882792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 354.902849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 358.864631] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 369.013139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 369.180857] Lustre: 14955:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 369.234458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 369.243659] Lustre: Skipped 1 previous similar message [ 369.316922] Lustre: lustre-MDT0001: new disk, initializing [ 369.455406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 369.499249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 369.520791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 373.573850] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 380.876769] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 385.364703] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 392.983430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 393.239751] Lustre: lustre-OST0000: new disk, initializing [ 393.244083] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 393.251078] Lustre: 16698:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 393.397852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.661421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 394.672059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 394.772814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 399.508762] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 407.560809] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 412.362066] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 412.552919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 416.641709] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 416.917984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 420.329092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 420.345085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 420.366958] Lustre: Skipped 1 previous similar message [ 420.377681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 420.380501] Lustre: Skipped 1 previous similar message [ 425.179193] Lustre: server umount lustre-OST0000 complete [ 430.561162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 430.575740] Lustre: Skipped 2 previous similar messages [ 434.833970] Lustre: server umount lustre-MDT0000 complete [ 435.684519] LustreError: 14966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 435.703083] LustreError: 14966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 438.381737] LustreError: 16697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230233 with bad export cookie 13350388949191371055 [ 438.384219] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 438.396211] LustreError: 16697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 438.772638] Lustre: server umount lustre-MDT0001 complete [ 443.399615] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 22:10:36 (1781230236) [ 445.769167] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 447.757523] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 22:10:40 (1781230240) [ 454.367491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 455.007689] LustreError: 18962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 455.065951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 459.606463] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 464.364274] LustreError: 18963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 464.380856] LustreError: 18963:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 467.846674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.182634] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 477.813803] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 482.147612] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 489.371165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 495.737175] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 500.841512] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 508.385752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 508.396829] LustreError: Skipped 1 previous similar message [ 508.398705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.418613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 508.423112] Lustre: Skipped 2 previous similar messages [ 510.442645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 514.251239] Lustre: server umount lustre-OST0000 complete [ 519.650980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 519.662246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.671038] Lustre: Skipped 1 previous similar message [ 523.784615] Lustre: server umount lustre-MDT0000 complete [ 525.793955] LustreError: 19673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.814759] LustreError: 19673:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 527.100150] LustreError: 18942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230321 with bad export cookie 13350388949191371930 [ 527.112808] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 527.122489] LustreError: 18942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 527.452103] Lustre: server umount lustre-MDT0001 complete [ 538.127644] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 541.027311] Key type lgssc unregistered [ 541.524669] LNet: 22137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 541.536301] LNetError: 22137:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 541.554919] LNet: Removed LNI 192.168.203.155@tcp [ 542.549167] Key type .llcrypt unregistered [ 542.554716] Key type ._llcrypt unregistered [ 567.408599] Key type ._llcrypt registered [ 567.411322] Key type .llcrypt registered [ 567.489510] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 568.643357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 568.673486] alg: No test for adler32 (adler32-zlib) [ 569.718372] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 569.892216] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 571.543226] Key type lgssc registered [ 572.693707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 580.750988] loop: module loaded [ 580.870301] loop0: detected capacity change from 0 to 204800000 [ 581.090628] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 581.113425] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 581.578645] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 587.957146] loop0: detected capacity change from 0 to 204800000 [ 588.133563] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 588.150860] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 588.162672] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 593.353426] loop0: detected capacity change from 0 to 204800000 [ 599.571940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 599.594946] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 600.983157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 601.043422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 601.167493] Lustre: lustre-MDT0000: new disk, initializing [ 601.274686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.294828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 605.649454] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 614.526228] loop1: detected capacity change from 0 to 204800000 [ 621.207742] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 621.569708] Lustre: lustre-OST07c6: new disk, initializing [ 621.581254] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 621.587989] Lustre: Skipped 1 previous similar message [ 621.591963] Lustre: 25289:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 621.672576] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 628.498210] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 630.390590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 630.402406] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 630.422181] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 633.693916] Lustre: 24294:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 633.701225] Lustre: 24294:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 633.705678] Lustre: 24294:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 633.710939] Lustre: 24294:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 633.716980] Lustre: 24294:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 633.725892] Lustre: 24294:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 649.704186] Lustre: 24293:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 649.716601] Lustre: 24293:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 649.723228] Lustre: 24293:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 649.732329] Lustre: 24293:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 649.739952] Lustre: 24293:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 649.747940] Lustre: 24293:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 655.023318] Lustre: 24295:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 655.041946] Lustre: 24295:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 655.048674] Lustre: 24295:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 655.054793] Lustre: 24295:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 655.063480] Lustre: 24295:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 655.072248] Lustre: 24295:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 655.085445] Lustre: 24295:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 655.091401] Lustre: 24295:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 655.100145] Lustre: 24295:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 655.108556] Lustre: 24295:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 655.115652] Lustre: 24295:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 655.122227] Lustre: 24295:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 658.433274] Lustre: 25855:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 658.444975] Lustre: 25855:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 658.453531] Lustre: 25855:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 658.460418] Lustre: 25855:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 658.469950] Lustre: 25855:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 658.486249] Lustre: 25855:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 658.507769] Lustre: 25855:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 658.517921] Lustre: 25855:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 658.528935] Lustre: 25855:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 658.542439] Lustre: 25855:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 658.555799] Lustre: 25855:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 658.570281] Lustre: 25855:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 664.176839] Lustre: 24293:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 664.198638] Lustre: 24293:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 664.204501] Lustre: 24293:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 664.213808] Lustre: 24293:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 664.220592] Lustre: 24293:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 664.229261] Lustre: 24293:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 664.236508] Lustre: 24293:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 664.243916] Lustre: 24293:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 664.256806] Lustre: 24293:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 664.274661] Lustre: 24293:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 664.288646] Lustre: 24293:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 664.301870] Lustre: 24293:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 672.453214] Lustre: 24293:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 672.463320] Lustre: 24293:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 672.468863] Lustre: 24293:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 672.479433] Lustre: 24293:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 672.502669] Lustre: 24293:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 672.510640] Lustre: 24293:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 672.519109] Lustre: 24293:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 672.527913] Lustre: 24293:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 672.552699] Lustre: 24293:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 672.564735] Lustre: 24293:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 672.575376] Lustre: 24293:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 672.589633] Lustre: 24293:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 727.468565] Lustre: server umount lustre-OST07c6 complete [ 747.194831] Lustre: server umount lustre-MDT0000 complete [ 764.727504] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 767.491957] Key type lgssc unregistered [ 767.837608] LNet: 27818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.859564] LNetError: 27818:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 767.888301] LNet: Removed LNI 192.168.203.155@tcp [ 768.598493] Key type .llcrypt unregistered [ 768.600904] Key type ._llcrypt unregistered [ 786.828725] Key type ._llcrypt registered [ 786.831547] Key type .llcrypt registered [ 786.916829] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 788.193695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 788.219572] alg: No test for adler32 (adler32-zlib) [ 789.227612] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 789.406748] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 791.135189] Key type lgssc registered [ 792.379485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 801.152094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 801.194887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 802.736764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 808.008940] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 813.032338] LustreError: 29072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 813.046394] LustreError: 29072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 818.146296] LustreError: 29073:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 818.802633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.376710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 823.997854] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 829.911617] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 833.871847] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 841.159755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 841.560501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 847.743442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 848.163200] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 854.151599] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 873.442217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 873.449490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 873.492810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 875.989642] Lustre: server umount lustre-OST0000 complete [ 880.611256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 880.618463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.631180] Lustre: Skipped 1 previous similar message [ 880.638438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 880.645603] Lustre: Skipped 1 previous similar message [ 883.685711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 883.690736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 886.131155] Lustre: server umount lustre-MDT0000 complete [ 888.804589] LustreError: 29072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 888.823816] LustreError: 29072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 889.847274] LustreError: 29054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230684 with bad export cookie 9415102827104418331 [ 889.857169] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 889.869700] LustreError: 29054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 890.218563] Lustre: server umount lustre-MDT0001 complete [ 901.488711] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 904.312386] Key type lgssc unregistered [ 904.599424] LNet: 32452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.608126] LNetError: 32452:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.625886] LNet: Removed LNI 192.168.203.155@tcp [ 905.578266] Key type .llcrypt unregistered [ 905.590830] Key type ._llcrypt unregistered [ 924.310954] Key type ._llcrypt registered [ 924.314819] Key type .llcrypt registered [ 924.438526] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 925.472346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 925.549787] alg: No test for adler32 (adler32-zlib) [ 926.729542] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 927.121361] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 928.807875] Key type lgssc registered [ 930.106087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 938.101977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 938.134911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 939.688894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 943.107784] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 950.245645] LustreError: 33705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 950.256678] LustreError: 33705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 951.773300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.262800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 956.881732] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 962.992433] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 967.307856] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 974.657486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 975.043119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 981.205764] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 984.311476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 986.810650] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 997.856647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 997.866849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.874996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 999.906090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 999.910525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 999.931521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.621908] Lustre: server umount lustre-OST0000 complete [ 1005.031374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.038599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1005.042754] Lustre: Skipped 1 previous similar message [ 1005.060058] Lustre: Skipped 2 previous similar messages [ 1010.148649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1010.156548] Lustre: Skipped 1 previous similar message [ 1015.276350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1015.292474] Lustre: Skipped 1 previous similar message [ 1019.359232] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1019.526159] Lustre: server umount lustre-MDT0000 complete [ 1020.391605] LustreError: 33700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1020.413237] LustreError: 33700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1023.966077] LustreError: 33686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230818 with bad export cookie 610868464270807224 [ 1023.980385] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1024.332380] Lustre: server umount lustre-MDT0001 complete [ 1036.060695] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1039.592771] Key type lgssc unregistered [ 1039.968621] LNet: 36885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1039.982661] LNetError: 36885:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1040.000585] LNet: Removed LNI 192.168.203.155@tcp [ 1040.968384] Key type .llcrypt unregistered [ 1040.976442] Key type ._llcrypt unregistered [ 1065.449471] Key type ._llcrypt registered [ 1065.455461] Key type .llcrypt registered [ 1065.594480] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1066.659827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1066.710413] alg: No test for adler32 (adler32-zlib) [ 1067.855786] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 1068.058686] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1069.727211] Key type lgssc registered [ 1070.990097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1078.784069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1078.804701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1080.438286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1084.810711] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1091.044160] LustreError: 38284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1091.070299] LustreError: 38284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1093.545831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.950663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1097.926471] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1103.635789] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1107.872689] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1115.430759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1115.777598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1122.196679] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1126.014484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1128.016243] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1135.654706] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1136.619453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.627705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1136.635957] Lustre: Skipped 2 previous similar messages [ 1136.650188] Lustre: Skipped 2 previous similar messages [ 1140.742359] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1141.970118] Lustre: server umount lustre-MDT0000 complete [ 1145.868977] LustreError: 38281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1146.136533] LustreError: 38266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230940 with bad export cookie 13065826032917263685 [ 1146.146604] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1146.849610] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.872226] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1146.875123] Lustre: Skipped 3 previous similar messages [ 1150.980870] LustreError: 39484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1151.004932] LustreError: 39484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1151.011912] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 1152.780582] Lustre: server umount lustre-MDT0001 complete [ 1174.460175] Lustre: server umount lustre-OST0000 complete [ 1193.288706] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1196.580830] Key type lgssc unregistered [ 1196.982621] LNet: 41794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1196.992587] LNetError: 41794:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1197.015919] LNet: Removed LNI 192.168.203.155@tcp [ 1198.152806] Key type .llcrypt unregistered [ 1198.154605] Key type ._llcrypt unregistered [ 1222.267860] Key type ._llcrypt registered [ 1222.271746] Key type .llcrypt registered [ 1222.365353] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1223.751370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1223.774984] alg: No test for adler32 (adler32-zlib) [ 1224.786655] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 1225.102861] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1226.759332] Key type lgssc registered [ 1228.101330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1236.275414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1236.289700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.983473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.679444] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1248.225175] LustreError: 43194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.249749] LustreError: 43194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1252.913044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1253.354459] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1253.523663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1258.968658] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1265.032183] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1269.385411] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1277.180952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1277.594049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.742223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1284.362342] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1291.130377] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1299.426871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1299.444868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1299.476273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1303.527335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1303.532224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1303.536476] Lustre: Skipped 1 previous similar message [ 1303.559370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.644104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1308.654559] Lustre: Skipped 3 previous similar messages [ 1313.247137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1313.437950] Lustre: server umount lustre-OST0000 complete [ 1334.243634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.258704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1334.266741] Lustre: Skipped 1 previous similar message [ 1334.284184] Lustre: Skipped 2 previous similar messages [ 1339.226381] Lustre: server umount lustre-MDT0000 complete [ 1343.457042] LustreError: 43189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1343.485725] LustreError: 43189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1343.741165] LustreError: 43175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231138 with bad export cookie 9886882979897447787 [ 1343.749704] LustreError: 43175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1343.751417] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1358.303370] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1358.573675] Lustre: server umount lustre-MDT0001 complete [ 1370.807204] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1373.517724] Key type lgssc unregistered [ 1373.698619] LNet: 46550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.708900] LNetError: 46550:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1373.735114] LNet: Removed LNI 192.168.203.155@tcp [ 1374.604982] Key type .llcrypt unregistered [ 1374.607905] Key type ._llcrypt unregistered [ 1401.612795] Key type ._llcrypt registered [ 1401.614955] Key type .llcrypt registered [ 1401.754774] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1402.890413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1403.025857] alg: No test for adler32 (adler32-zlib) [ 1404.121979] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 1404.367450] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1406.032055] Key type lgssc registered [ 1407.353755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1415.666754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1415.701865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.254421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1422.181405] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1426.410830] LustreError: 47951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.428202] LustreError: 47951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1431.522517] LustreError: 47950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1431.619688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1431.836957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1438.010138] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1445.521482] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1450.865329] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1458.585710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1459.195319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.286474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1466.549906] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1473.010193] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1481.419645] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1485.011243] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1496.872087] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-1122h28m09s [ 1498.444553] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1503.200678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1503.206026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1503.215740] Lustre: Skipped 2 previous similar messages [ 1503.225916] Lustre: Skipped 2 previous similar messages [ 1504.291673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1505.298518] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1505.301590] Lustre: Skipped 2 previous similar messages [ 1508.731848] Lustre: server umount lustre-MDT0000 complete [ 1510.372167] LustreError: 47950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1510.400823] LustreError: 47950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1512.453702] LustreError: 47945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1512.954397] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1516.524051] LustreError: 47947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1516.553295] LustreError: 47947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1520.328212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1520.640465] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1520.897642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.005688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1521.901238] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1526.430484] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1531.900683] Lustre: DEBUG MARKER: Wait for df (36061) ... [ 1535.090714] Lustre: DEBUG MARKER: done [ 1541.600068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1541.609526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1541.635449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1541.653725] Lustre: Skipped 1 previous similar message [ 1542.631182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1547.669470] Lustre: server umount lustre-OST0000 complete [ 1552.867656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1552.883967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1552.884956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1552.899513] Lustre: Skipped 3 previous similar messages [ 1559.026651] Lustre: server umount lustre-MDT0000 complete [ 1560.033850] LustreError: 47950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1560.068053] LustreError: 47950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1562.965404] LustreError: 47932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231357 with bad export cookie 12498648040515429808 [ 1562.979499] LustreError: 47932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1562.987444] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1563.411271] Lustre: server umount lustre-MDT0001 complete [ 1576.765365] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1580.169666] Key type lgssc unregistered [ 1580.504079] LNet: 52503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1580.534702] LNetError: 52503:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1580.550923] LNet: Removed LNI 192.168.203.155@tcp [ 1581.495206] Key type .llcrypt unregistered [ 1581.497643] Key type ._llcrypt unregistered [ 1625.562287] Key type ._llcrypt registered [ 1625.568299] Key type .llcrypt registered [ 1625.752723] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1627.201207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1627.246061] alg: No test for adler32 (adler32-zlib) [ 1628.410602] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 1628.687359] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1630.327785] Key type lgssc registered [ 1631.496608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1640.053405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1640.072651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1641.427757] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1641.513437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1641.748664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1646.042412] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1655.539527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1655.718701] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1655.772778] Lustre: 54696:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1656.111974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1661.348558] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1667.647464] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1672.122742] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1680.787892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1681.282737] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1681.636827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.042216] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1689.307746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1693.777255] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1700.580562] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-1122h31m33s [ 1702.305747] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1710.191904] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1712.965486] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1713.036749] LustreError: 54704:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1733.184219] Lustre: lustre-MDT0000: Client ac2b967c-3244-4282-9956-2b8d5eef43a6 (at 192.168.203.55@tcp) reconnecting [ 1753.049260] LustreError: 54704:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1755.787690] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1758.598665] Lustre: DEBUG MARKER: done [ 1766.368448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1766.372526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1766.395740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1771.006269] Lustre: server umount lustre-OST0000 complete [ 1776.608160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1776.611153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1776.626344] Lustre: Skipped 2 previous similar messages [ 1776.632574] Lustre: Skipped 1 previous similar message [ 1777.702083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1777.709956] Lustre: Skipped 2 previous similar messages [ 1781.003325] Lustre: server umount lustre-MDT0000 complete [ 1785.510093] LustreError: 54688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231580 with bad export cookie 7454197928866054326 [ 1785.529190] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1785.541757] LustreError: 54688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1785.861083] Lustre: server umount lustre-MDT0001 complete [ 1798.509699] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1802.310920] Key type lgssc unregistered [ 1802.710083] LNet: 58667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1802.720538] LNetError: 58667:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1802.749048] LNet: Removed LNI 192.168.203.155@tcp [ 1803.547942] Key type .llcrypt unregistered [ 1803.551103] Key type ._llcrypt unregistered [ 1848.125242] Key type ._llcrypt registered [ 1848.131522] Key type .llcrypt registered [ 1848.277083] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1849.376647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1849.414653] alg: No test for adler32 (adler32-zlib) [ 1850.664762] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 1850.913529] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1852.591172] Key type lgssc registered [ 1853.915738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1857.399509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1857.423522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1858.759162] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1858.799068] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1859.076497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1860.196687] Lustre: Failing over lustre-MDT0000 [ 1860.739815] Lustre: server umount lustre-MDT0000 complete [ 1869.354356] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 22:34:21 (1781231661) [ 1876.897080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.348470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.866668] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1890.617587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1890.715542] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1890.736656] Lustre: 61449:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1891.082799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1895.760781] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1901.381872] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1905.249582] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1912.839597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1913.017247] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1913.235940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1920.473477] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1922.300061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1926.866411] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1932.657740] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1942.496692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1942.502795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1942.515425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1943.011425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.027440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1943.030268] Lustre: Skipped 1 previous similar message [ 1946.993969] Lustre: server umount lustre-MDT0000 complete [ 1951.184093] LustreError: 61480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231745 with bad export cookie 13076291939595477924 [ 1951.194215] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1951.197283] LustreError: 61480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1951.686553] Lustre: server umount lustre-MDT0001 complete [ 1955.589670] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1960.951668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1968.609209] Lustre: 60372:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781231747/real 1781231747] req@00000000ea22a047 x1867756753427712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781231763 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1968.654716] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1968.681549] Lustre: Skipped 1 previous similar message [ 1969.839135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.679224] Lustre: 60373:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781231750/real 1781231750] req@00000000e63873ed x1867756753427840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781231766 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1974.751214] Lustre: 60371:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781231753/real 1781231753] req@00000000b1562cc2 x1867756753427968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781231769 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1975.847239] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb578523ce6e46c87 [ 1975.869135] Lustre: MGC192.168.203.155@tcp: Connection restored to 0@lo (at 0@lo) [ 1976.228777] LustreError: 63074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1976.251379] LustreError: 63074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1976.356925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1976.456782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1977.311162] Lustre: 60371:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781231755/real 1781231755] req@000000005fb3bb38 x1867756753428224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781231771 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1977.379068] LustreError: 63075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1978.405600] LustreError: 63074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1978.425397] LustreError: 63074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1980.918922] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1981.473070] Lustre: 60373:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781231760/real 1781231760] req@000000005362b542 x1867756753428352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781231776 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1983.459142] LustreError: 64881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1983.494384] LustreError: 64881:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1988.579612] LustreError: 63075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.610065] LustreError: 63075:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1991.168072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.599951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1996.379412] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1996.797219] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1997.873671] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2002.884109] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2008.252364] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2017.248973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2017.262390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.291280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2022.067498] Lustre: server umount lustre-MDT0000 complete [ 2023.393952] LustreError: 65601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2023.423247] LustreError: 65601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2025.658594] LustreError: 61480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231820 with bad export cookie 13076291939595480199 [ 2025.668577] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2025.680545] LustreError: 61480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2026.119826] Lustre: server umount lustre-MDT0001 complete [ 2032.377549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2041.335919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2044.895263] Lustre: 60373:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781231823/real 1781231823] req@00000000086e68eb x1867756753456128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781231839 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2044.918654] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.938913] Lustre: Skipped 2 previous similar messages [ 2051.106726] LustreError: 60369:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004eb46ba2 x1867756753457152/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2051.499994] LustreError: 63075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2051.614558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2051.691766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2055.695536] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2064.914365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2069.952726] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2070.506764] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2076.002778] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2079.854451] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2086.291911] Lustre: server umount lustre-MDT0000 complete [ 2088.942917] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.951251] LustreError: 65601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2088.963364] Lustre: Skipped 1 previous similar message [ 2088.994551] LustreError: 65601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2090.065223] LustreError: 61480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231884 with bad export cookie 13076291939595481578 [ 2090.066981] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2090.073431] LustreError: 61480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2090.399904] Lustre: server umount lustre-MDT0001 complete [ 2096.966516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2100.967670] Lustre: DEBUG MARKER: files compared the same [ 2112.553233] Lustre: server umount lustre-OST0000 complete [ 2130.353616] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2133.504304] Key type lgssc unregistered [ 2133.821310] LNet: 70370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2133.832739] LNetError: 70370:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2133.841878] LNet: Removed LNI 192.168.203.155@tcp [ 2134.720261] Key type .llcrypt unregistered [ 2134.725915] Key type ._llcrypt unregistered [ 2158.622316] Key type ._llcrypt registered [ 2158.627841] Key type .llcrypt registered [ 2158.780140] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2159.968416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2160.054237] alg: No test for adler32 (adler32-zlib) [ 2161.117642] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 2161.486856] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2163.199201] Key type lgssc registered [ 2164.404258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2171.731437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2171.763431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2173.349507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2177.690158] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2183.651645] LustreError: 71772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2183.674416] LustreError: 71772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2186.159136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.393979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2190.692620] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2195.848776] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2199.595134] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2206.310991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2206.741584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2213.313114] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2219.073788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2220.460642] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2232.290551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2232.294208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2232.300737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2234.338064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2234.338606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2234.359961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.040280] Lustre: server umount lustre-OST0000 complete [ 2243.040476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2243.048220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.062366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2243.070133] Lustre: Skipped 1 previous similar message [ 2249.699347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2249.707418] Lustre: Skipped 3 previous similar messages [ 2254.303111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2254.544374] Lustre: server umount lustre-MDT0000 complete [ 2254.815985] LustreError: 71772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.836740] LustreError: 71772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2258.476424] LustreError: 72490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232053 with bad export cookie 8799661586762936141 [ 2258.488646] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2258.489365] LustreError: 72490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2258.939253] Lustre: server umount lustre-MDT0001 complete [ 2269.828563] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2272.568736] Key type lgssc unregistered [ 2272.948858] LNet: 74951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2272.963436] LNetError: 74951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2274.026971] LNet: Removed LNI 192.168.203.155@tcp [ 2274.888985] Key type .llcrypt unregistered [ 2274.894472] Key type ._llcrypt unregistered [ 2296.777622] Key type ._llcrypt registered [ 2296.780920] Key type .llcrypt registered [ 2296.875848] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2297.563049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2297.604402] alg: No test for adler32 (adler32-zlib) [ 2298.663449] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 2298.885525] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2300.591177] Key type lgssc registered [ 2301.493068] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2308.469088] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2308.500159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2324.064227] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2329.183325] LustreError: 76325:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 2356.127353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2363.804455] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2370.947293] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2380.252547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2380.417609] Lustre: *** cfs_fail_loc=706, val=0*** [ 2380.937882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2381.010611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2386.415623] LustreError: 76350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.417358] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2386.446551] LustreError: 76350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2390.501794] LustreError: 76351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2390.514140] LustreError: 76351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2395.636458] LustreError: 77299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.970128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.423981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2401.211251] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2407.087464] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2410.983179] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2416.610457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2416.618149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2416.633872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2417.136797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2417.155618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2417.161463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2417.179474] Lustre: Skipped 1 previous similar message [ 2422.241913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2427.359340] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2427.361324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2427.373033] Lustre: Skipped 1 previous similar message [ 2427.497366] Lustre: server umount lustre-OST0000 complete [ 2432.482660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.482723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2432.517269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2432.551907] Lustre: Skipped 2 previous similar messages [ 2437.608488] LustreError: 77302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2437.622274] LustreError: 77302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2437.702778] Lustre: server umount lustre-MDT0000 complete [ 2441.982693] LustreError: 76333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232236 with bad export cookie 590034297561112326 [ 2442.000651] LustreError: 76333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2442.005838] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2442.668579] Lustre: server umount lustre-MDT0001 complete [ 2454.952822] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2458.475396] Key type lgssc unregistered [ 2458.725643] LNet: 79583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2458.735594] LNetError: 79583:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2458.757624] LNet: Removed LNI 192.168.203.155@tcp [ 2459.568928] Key type .llcrypt unregistered [ 2459.573532] Key type ._llcrypt unregistered [ 2487.645303] Key type ._llcrypt registered [ 2487.654193] Key type .llcrypt registered [ 2487.751980] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2488.845492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2488.935814] alg: No test for adler32 (adler32-zlib) [ 2490.112272] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 2490.462489] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2492.119259] Key type lgssc registered [ 2493.796202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2503.897548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2503.917894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2510.062951] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2519.033370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2519.834429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2525.161223] LustreError: 81656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2525.195070] LustreError: 81656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2525.204885] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2530.276663] LustreError: 81660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.288504] LustreError: 81660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2534.596671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2535.083827] LustreError: 81656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2535.104252] LustreError: 81656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2535.125133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2540.513927] LustreError: 81659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.536711] LustreError: 81659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2541.530503] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2545.638811] LustreError: 81660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.658562] LustreError: 81660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2550.510667] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2550.521077] Lustre: MGS: Client 68465ba4-e4b5-4756-aa58-de596dc77e5a (at 0@lo) reconnecting [ 2550.553586] LustreError: 81670:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2550.768377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2550.772475] Lustre: Skipped 2 previous similar messages [ 2550.916121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2551.012602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2555.379701] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2571.234765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2571.244308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2571.258562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2571.751350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2575.215415] Lustre: server umount lustre-OST0000 complete [ 2581.984409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2581.987424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2582.000936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2582.012040] Lustre: Skipped 2 previous similar messages [ 2585.807042] Lustre: server umount lustre-MDT0000 complete [ 2587.111978] LustreError: 83462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2587.143855] LustreError: 83462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2590.074855] Lustre: server umount lustre-MDT0001 complete [ 2594.318749] Lustre: server umount MGS complete [ 2606.080497] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:46:39 (1781232399) [ 2638.128975] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 2650.240995] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2691.975171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2696.842968] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2706.895323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2707.151556] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2707.199799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2707.284409] Lustre: lustre-MDT0001: new disk, initializing [ 2707.371790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2707.407910] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2707.421546] Lustre: 88726:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2707.430519] LustreError: 88726:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2712.085752] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2723.569469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2723.845935] Lustre: lustre-OST0000: new disk, initializing [ 2723.849800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2723.856867] Lustre: 89475:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2723.949584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2730.638729] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2732.640832] Lustre: 89862:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000815211b8 x1867757424516224/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781232543 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2738.868066] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2745.786601] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2745.794788] Lustre: MGS: Client c2276fad-005d-43cc-a4bc-42992f4db154 (at 0@lo) reconnecting [ 2745.835183] Lustre: 90408:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2745.871930] LustreError: 88712:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2745.883344] LustreError: 88712:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2745.894781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2745.994849] Lustre: lustre-MDT0000: new disk, initializing [ 2746.163203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2746.206419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2746.395991] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2747.316531] Lustre: 90432:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000036acc73d x1867757424523008/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1781232558 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2750.603770] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2752.427062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2752.437147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2752.581835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2777.568803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2777.570809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2777.599461] Lustre: Skipped 2 previous similar messages [ 2777.618809] Lustre: Skipped 1 previous similar message [ 2781.748406] Lustre: server umount lustre-OST0000 complete [ 2787.808079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2787.812136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2787.828610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2787.848851] Lustre: Skipped 1 previous similar message [ 2792.177899] Lustre: server umount lustre-MDT0000 complete [ 2792.932950] LustreError: 90701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2792.961128] LustreError: 90701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2802.678017] Lustre: server umount lustre-MDT0001 complete [ 2806.748158] Lustre: server umount MGS complete [ 2820.328342] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 22:50:13 (1781232613) [ 2841.014388] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2843.729872] Key type lgssc unregistered [ 2844.057809] LNet: 92894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2844.062833] LNetError: 92894:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2844.088352] LNet: Removed LNI 192.168.203.155@tcp [ 2844.979826] Key type .llcrypt unregistered [ 2844.982684] Key type ._llcrypt unregistered [ 2863.204872] Key type ._llcrypt registered [ 2863.209210] Key type .llcrypt registered [ 2863.346480] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2865.158966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2865.276929] alg: No test for adler32 (adler32-zlib) [ 2866.460793] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 2866.706062] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2868.417764] Key type lgssc registered [ 2869.855689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2879.072076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2879.130143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.035199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.420484] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2891.235426] LustreError: 94102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2891.274463] LustreError: 94102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2894.413080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.774612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2899.242516] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2904.801250] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2909.320669] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2917.242562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2917.583162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2923.996463] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2925.885447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2930.665047] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2940.399723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2940.411735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.431184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2941.412099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2941.412989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2941.446102] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2946.541841] LustreError: 96253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.564568] LustreError: 96253:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2946.665505] Lustre: server umount lustre-OST0000 complete [ 2951.137876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2951.145917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2951.173937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2951.177121] Lustre: Skipped 1 previous similar message [ 2956.770880] LustreError: 95258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.797854] LustreError: 95258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2956.823287] Lustre: server umount lustre-MDT0000 complete [ 2960.896953] LustreError: 94081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232755 with bad export cookie 15748150702933707977 [ 2960.909751] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2961.473754] Lustre: server umount lustre-MDT0001 complete [ 2972.699756] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2975.338492] Key type lgssc unregistered [ 2975.704732] LNet: 97279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2975.714975] LNetError: 97279:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2976.741233] LNet: Removed LNI 192.168.203.155@tcp [ 2977.605424] Key type .llcrypt unregistered [ 2977.608753] Key type ._llcrypt unregistered [ 2994.813509] Key type ._llcrypt registered [ 2994.818697] Key type .llcrypt registered [ 2994.955232] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2996.089614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2996.173367] alg: No test for adler32 (adler32-zlib) [ 2997.318421] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 2997.588727] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2999.344797] Key type lgssc registered [ 3000.740291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3007.689775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3007.690393] LustreError: 98343:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3007.695537] Lustre: Skipped 1 previous similar message [ 3007.721636] LustreError: 98346:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3007.735620] LustreError: 98343:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4976 [ 3007.741888] LustreError: 98343:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3007.750386] LustreError: 98343:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3007.781187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3009.573988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3018.426861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.757282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3023.680970] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3029.209788] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing lsmod [ 3032.104731] LustreError: 99519:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3032.127697] LustreError: 99520:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3032.134338] LustreError: 99519:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4974 [ 3032.143701] LustreError: 99519:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3032.154627] LustreError: 99519:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3032.176188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3032.658753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3034.690805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3039.201894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3039.213771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3039.235958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3039.715795] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3039.725822] Lustre: Skipped 1 previous similar message [ 3041.877164] Lustre: server umount lustre-MDT0000 complete [ 3043.814379] LustreError: 98818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3043.831174] LustreError: 98818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3045.598209] LustreError: 98365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232840 with bad export cookie 5052320163349570246 [ 3045.602276] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3045.621941] LustreError: 98365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3045.904036] LustreError: 98815:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a7c9b969 x1867757956125440/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3046.093457] Lustre: server umount lustre-MDT0001 complete [ 3059.761094] Lustre: server umount lustre-OST0000 complete [ 3068.667995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.267786] LustreError: 100584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3069.393358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3073.731970] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3074.536242] LustreError: 100585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3079.653365] LustreError: 100584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3083.213615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3083.493957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3087.612708] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3093.864494] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3097.834085] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3104.647893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3105.066569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3112.110358] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3114.298960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3118.813339] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3129.837789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3129.838770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.848242] LustreError: Skipped 1 previous similar message [ 3129.856992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3129.879522] Lustre: Skipped 1 previous similar message [ 3129.888364] Lustre: Skipped 2 previous similar messages [ 3133.390489] Lustre: server umount lustre-OST0000 complete [ 3140.066620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3140.067509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.071454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3140.091793] Lustre: Skipped 1 previous similar message [ 3140.104477] Lustre: Skipped 1 previous similar message [ 3144.010927] Lustre: server umount lustre-MDT0000 complete [ 3145.191032] LustreError: 100584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3145.218311] LustreError: 100584:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3148.053860] LustreError: 100566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232942 with bad export cookie 5052320163349571177 [ 3148.062778] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3148.067795] LustreError: 100566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3148.511654] Lustre: server umount lustre-MDT0001 complete [ 3159.536928] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3162.213574] Key type lgssc unregistered [ 3162.438623] LNet: 103762:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3162.459794] LNetError: 103762:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3162.482685] LNet: Removed LNI 192.168.203.155@tcp [ 3163.807896] Key type .llcrypt unregistered [ 3163.812609] Key type ._llcrypt unregistered [ 3190.967835] Key type ._llcrypt registered [ 3190.975442] Key type .llcrypt registered [ 3191.112501] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3192.234197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3192.319281] alg: No test for adler32 (adler32-zlib) [ 3193.460458] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 3193.734719] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3195.483307] Key type lgssc registered [ 3196.552576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3204.139879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3204.159119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3205.570956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3210.530476] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3215.845135] LustreError: 105165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3215.885800] LustreError: 105165:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3220.475961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.879728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3225.094290] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3231.380031] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3235.774579] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3243.249519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3243.611831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3250.280533] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3250.806546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3256.172195] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3262.615783] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3266.908760] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3276.771538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3276.772478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3276.787288] LustreError: Skipped 1 previous similar message [ 3276.789388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3276.814610] Lustre: Skipped 1 previous similar message [ 3278.354407] Lustre: server umount lustre-OST0000 complete [ 3287.013574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3287.017505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3287.033965] Lustre: Skipped 1 previous similar message [ 3287.053178] Lustre: Skipped 2 previous similar messages [ 3288.773412] Lustre: server umount lustre-MDT0000 complete [ 3292.130045] LustreError: 105164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3292.144471] LustreError: 105164:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3292.818810] LustreError: 107357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233087 with bad export cookie 4032088877227564123 [ 3292.827832] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3292.830752] LustreError: 107357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3293.260891] Lustre: server umount lustre-MDT0001 complete [ 3305.514855] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3308.575331] Key type lgssc unregistered [ 3308.852350] LNet: 108446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3308.858217] LNetError: 108446:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3308.877287] LNet: Removed LNI 192.168.203.155@tcp [ 3309.795114] Key type .llcrypt unregistered [ 3309.800430] Key type ._llcrypt unregistered [ 3329.418351] Key type ._llcrypt registered [ 3329.424377] Key type .llcrypt registered [ 3329.552620] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3330.717080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3330.792449] alg: No test for adler32 (adler32-zlib) [ 3331.982787] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 3332.302293] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3334.031925] Key type lgssc registered [ 3335.484752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3344.303145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3344.337392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3346.034918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3351.023447] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3355.618380] LustreError: 109651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3355.635672] LustreError: 109651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3360.736788] LustreError: 109652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3361.682139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3362.147967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3366.696773] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3373.239718] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3377.487543] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3384.307208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3384.631362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3390.786032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3391.007532] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3397.053766] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3406.308840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.311822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3406.333506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3406.347558] Lustre: Skipped 1 previous similar message [ 3411.426809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3411.437718] Lustre: Skipped 1 previous similar message [ 3416.561463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3416.568203] Lustre: Skipped 1 previous similar message [ 3420.127119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3420.284651] Lustre: server umount lustre-OST0000 complete [ 3426.786593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3426.794208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3426.825282] Lustre: Skipped 2 previous similar messages [ 3426.842222] Lustre: Skipped 1 previous similar message [ 3431.176800] Lustre: server umount lustre-MDT0000 complete [ 3431.906096] LustreError: 109651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3431.921128] LustreError: 109651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3435.185648] LustreError: 111891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233229 with bad export cookie 9831287692685499343 [ 3435.186834] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3435.193024] LustreError: 111891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3435.534117] Lustre: server umount lustre-MDT0001 complete [ 3446.130936] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3448.691708] Key type lgssc unregistered [ 3448.931311] LNet: 112831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3448.940886] LNetError: 112831:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3448.956761] LNet: Removed LNI 192.168.203.155@tcp [ 3449.750587] Key type .llcrypt unregistered [ 3449.752433] Key type ._llcrypt unregistered [ 3474.081371] Key type ._llcrypt registered [ 3474.083868] Key type .llcrypt registered [ 3474.199205] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3475.350218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3475.380320] alg: No test for adler32 (adler32-zlib) [ 3476.473799] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 3476.720716] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3478.479194] Key type lgssc registered [ 3479.874818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3488.656860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3488.679646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3490.207662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3495.174521] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3499.513298] LustreError: 114245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3499.545269] LustreError: 114245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3504.616589] LustreError: 114246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3504.808247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.241786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3510.030973] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3515.782603] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3519.807555] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3528.531022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3528.990812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3535.170249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3535.870954] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3542.064110] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3549.146976] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3552.663651] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3552.668099] Lustre: Skipped 2 previous similar messages [ 3555.910548] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3555.959607] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3556.886287] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3556.890724] Lustre: Skipped 2 previous similar messages [ 3562.067636] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3562.072422] Lustre: Skipped 1 previous similar message [ 3567.389658] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.55@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3567.409722] Lustre: Skipped 2 previous similar messages [ 3569.865234] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.55@tcp 192.168.0.[2,10]@tcp [ 3579.953315] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.55/24@tcp in log lustre-MDT0000 [ 3579.960518] Lustre: Skipped 2 previous similar messages [ 3589.703755] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.55/24@tcp [ 3589.713410] Lustre: Skipped 1 previous similar message [ 3596.768567] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3596.768858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3596.788265] LustreError: Skipped 1 previous similar message [ 3596.790400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3596.844765] Lustre: Skipped 1 previous similar message [ 3601.903959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3601.910489] Lustre: Skipped 2 previous similar messages [ 3602.278078] Lustre: server umount lustre-OST0000 complete [ 3607.528805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3607.543195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3607.576027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3612.129189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3612.138400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3613.475709] Lustre: server umount lustre-MDT0000 complete [ 3616.773572] LustreError: 116941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233411 with bad export cookie 15662047327692470577 [ 3616.779303] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3616.790685] LustreError: 116941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3617.040683] Lustre: server umount lustre-MDT0001 complete [ 3628.654958] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3631.279125] Key type lgssc unregistered [ 3631.594940] LNet: 118883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3631.605802] LNetError: 118883:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3631.622758] LNet: Removed LNI 192.168.203.155@tcp [ 3632.515815] Key type .llcrypt unregistered [ 3632.519655] Key type ._llcrypt unregistered [ 3661.800442] Key type ._llcrypt registered [ 3661.804317] Key type .llcrypt registered [ 3661.916259] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3663.348919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3663.411360] alg: No test for adler32 (adler32-zlib) [ 3664.484249] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 3664.768864] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3666.535485] Key type lgssc registered [ 3667.860558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3675.778685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3675.833738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.429868] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3677.468414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3677.481915] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.55/24@tcp [ 3681.725660] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3687.906154] LustreError: 120429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3687.921663] LustreError: 120429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3690.384267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3690.668656] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3690.686827] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.55/24@tcp [ 3690.732361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3695.052808] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3701.227683] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3705.097958] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3711.421561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3711.840657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3717.883123] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3724.023599] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3724.158825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3736.548947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3736.558167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3736.570937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3739.619645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3739.625275] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3739.636907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3741.405630] Lustre: server umount lustre-OST0000 complete [ 3744.737565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3744.744651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3744.747582] Lustre: Skipped 1 previous similar message [ 3744.764097] Lustre: Skipped 2 previous similar messages [ 3749.865553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3749.870577] Lustre: Skipped 1 previous similar message [ 3751.025739] Lustre: server umount lustre-MDT0000 complete [ 3754.791727] LustreError: 120409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233549 with bad export cookie 7896146614997410512 [ 3754.801537] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3755.117727] Lustre: server umount lustre-MDT0001 complete [ 3766.793261] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3770.037791] Key type lgssc unregistered [ 3770.320625] LNet: 123660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3770.328872] LNetError: 123660:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3770.358016] LNet: Removed LNI 192.168.203.155@tcp [ 3771.357084] Key type .llcrypt unregistered [ 3771.362172] Key type ._llcrypt unregistered [ 3799.936595] Key type ._llcrypt registered [ 3799.946522] Key type .llcrypt registered [ 3800.097509] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3816.679160] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3817.681257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3817.703336] alg: No test for adler32 (adler32-zlib) [ 3818.778222] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 3819.032225] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3820.743171] Key type lgssc registered [ 3822.080688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3859.958043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3860.011342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3861.344745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3861.396734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3861.516581] Lustre: lustre-MDT0000: new disk, initializing [ 3861.679153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3861.718193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3866.643796] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3877.535849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3877.660554] Lustre: 127029:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3877.717226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3877.722977] Lustre: Skipped 1 previous similar message [ 3877.819755] Lustre: lustre-MDT0001: new disk, initializing [ 3877.957150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3878.014375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3878.038726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3882.936468] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3890.292835] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3894.453595] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3901.543792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3901.937413] Lustre: lustre-OST0000: new disk, initializing [ 3901.942400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3901.948464] Lustre: 128770:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3902.078612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3908.165510] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3910.699676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3910.713800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3910.811419] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3916.298656] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3921.962821] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3922.249779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3927.403262] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3927.643783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3931.622544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3931.623382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3931.629944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3931.652680] Lustre: Skipped 1 previous similar message [ 3936.119438] Lustre: server umount lustre-OST0000 complete [ 3941.857620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3941.866052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3941.884713] Lustre: Skipped 1 previous similar message [ 3941.901108] Lustre: Skipped 2 previous similar messages [ 3946.268586] Lustre: server umount lustre-MDT0000 complete [ 3950.409107] LustreError: 127019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233745 with bad export cookie 8890323429369685167 [ 3950.412986] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3950.420107] LustreError: 127019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3950.842480] Lustre: server umount lustre-MDT0001 complete [ 3959.812357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3960.285853] LustreError: 130840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3960.316324] LustreError: 130840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3960.392881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3965.416359] LustreError: 130841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3965.735910] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3970.531763] LustreError: 130840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3974.432530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3974.920129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3979.933408] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3986.607724] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3990.854740] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3998.733649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3999.156785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4006.020283] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4011.534501] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4016.416879] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4016.681732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4021.706151] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4021.948229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4030.644704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4030.828215] Lustre: lustre-OST0001: new disk, initializing [ 4030.830718] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4030.835183] Lustre: 133837:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4030.920561] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4037.199313] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4040.313978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4040.329978] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4040.398067] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4045.470274] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4045.685238] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4050.322159] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4050.573811] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4055.295562] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97ef90037000.ost_server_uuid 50 [ 4057.096597] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff97ef90037000.ost_server_uuid in IDLE state after 0 sec [ 4071.395850] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4071.409483] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4071.418784] LustreError: Skipped 1 previous similar message [ 4071.425771] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4071.455787] Lustre: Skipped 1 previous similar message [ 4073.930223] Lustre: server umount lustre-OST0001 complete [ 4076.549507] LustreError: 133846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4076.564394] LustreError: 133846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4081.643076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4081.643755] LustreError: 132436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4081.653672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4081.653680] Lustre: Skipped 1 previous similar message [ 4081.663538] Lustre: Skipped 1 previous similar message [ 4081.702576] LustreError: 132436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4084.919461] Lustre: server umount lustre-OST0000 complete [ 4091.880062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4091.889606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4091.895033] Lustre: Skipped 1 previous similar message [ 4091.906799] Lustre: Skipped 2 previous similar messages [ 4095.537745] Lustre: server umount lustre-MDT0000 complete [ 4097.002257] LustreError: 130841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4097.024453] LustreError: 130841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4099.420138] LustreError: 130822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233894 with bad export cookie 8890323429369686049 [ 4099.425053] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4099.429881] LustreError: 130822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4099.784652] Lustre: server umount lustre-MDT0001 complete [ 4119.851091] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4122.575553] Key type lgssc unregistered [ 4122.879240] LNet: 136676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4122.893140] LNetError: 136676:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4122.905730] LNet: Removed LNI 192.168.203.155@tcp [ 4123.620715] Key type .llcrypt unregistered [ 4123.622541] Key type ._llcrypt unregistered [ 4163.342202] Key type ._llcrypt registered [ 4163.344488] Key type .llcrypt registered [ 4163.441063] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4178.994202] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4180.425544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4180.528742] alg: No test for adler32 (adler32-zlib) [ 4181.662148] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 4181.950141] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4183.631771] Key type lgssc registered [ 4185.037071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4219.979894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4220.033920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4221.237337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4221.262634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4221.323760] Lustre: lustre-MDT0000: new disk, initializing [ 4221.397983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4221.431549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4225.578842] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4236.298644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4236.500640] Lustre: 140815:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4236.580173] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4236.588151] Lustre: Skipped 1 previous similar message [ 4236.680385] Lustre: lustre-MDT0001: new disk, initializing [ 4236.764409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4236.804955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4236.817610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4240.947420] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4248.690662] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4252.957605] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4259.815131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4260.075652] Lustre: lustre-OST0000: new disk, initializing [ 4260.079703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4260.085711] Lustre: 142553:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4260.191525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4266.767417] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4269.154901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4269.175814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4269.265798] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4275.530786] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4280.943713] Lustre: Failing over lustre-OST0000 [ 4281.322160] Lustre: server umount lustre-OST0000 complete [ 4298.207628] Lustre: 139039:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781234076/real 1781234076] req@000000005532df17 x1867759197650560/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1781234092 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4298.245889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4300.319140] Lustre: 139038:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781234079/real 1781234079] req@000000000cfdc348 x1867759197651072/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781234095 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4300.319219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4300.342633] Lustre: 139038:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4302.765518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4303.274724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4303.297156] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4305.294363] Lustre: 139036:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781234084/real 1781234084] req@000000003e8514c9 x1867759197652864/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781234100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4305.347744] Lustre: 139036:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4308.385726] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4310.480917] Lustre: 139039:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781234089/real 1781234089] req@00000000f6c0f1a2 x1867759197654016/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781234105 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4310.501598] Lustre: 139039:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4310.563675] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4310.566487] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4310.975328] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4314.527219] Lustre: 139037:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781234093/real 1781234093] req@00000000386be1ee x1867759197654912/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781234109 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4314.542698] Lustre: 139037:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4315.170042] Lustre: Failing over lustre-MDT0000 [ 4315.399417] Lustre: server umount lustre-MDT0000 complete [ 4318.692485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4318.700043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4318.754341] Lustre: Skipped 2 previous similar messages [ 4323.812170] LustreError: 140825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4323.847103] LustreError: 140825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4328.940503] LustreError: 140824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4328.973446] LustreError: 140824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4333.933780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4334.055628] LustreError: 141941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4334.075733] LustreError: 141941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4334.102412] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4334.341946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4334.385227] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4339.028624] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4339.213739] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4339.700427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4339.706624] Lustre: Skipped 1 previous similar message [ 4339.719814] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4339.757670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4349.922488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.924986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4349.947959] Lustre: Skipped 1 previous similar message [ 4349.960408] Lustre: Skipped 1 previous similar message [ 4355.048842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4355.065052] Lustre: Skipped 1 previous similar message [ 4360.172356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4360.176733] Lustre: Skipped 1 previous similar message [ 4361.183110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4361.448076] Lustre: server umount lustre-OST0000 complete [ 4370.402078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4370.402661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4370.410180] Lustre: Skipped 1 previous similar message [ 4370.412238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4370.438720] Lustre: Skipped 1 previous similar message [ 4371.827475] Lustre: server umount lustre-MDT0000 complete [ 4375.520401] LustreError: 140824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4375.542150] LustreError: 140824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4376.092674] LustreError: 143697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234170 with bad export cookie 3052954578320573939 [ 4376.104839] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4376.115389] LustreError: 143697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4376.827207] Lustre: server umount lustre-MDT0001 complete [ 4389.389632] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4392.219548] Key type lgssc unregistered [ 4392.474937] LNet: 146046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4392.486923] LNetError: 146046:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4392.517713] LNet: Removed LNI 192.168.203.155@tcp [ 4393.484132] Key type .llcrypt unregistered [ 4393.487301] Key type ._llcrypt unregistered [ 4431.411693] Key type ._llcrypt registered [ 4431.418927] Key type .llcrypt registered [ 4431.523990] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4432.569044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4432.585912] alg: No test for adler32 (adler32-zlib) [ 4433.761561] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 4434.054984] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4435.743196] Key type lgssc registered [ 4437.014153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4445.828652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4445.841547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4447.446797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4452.729065] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4457.958117] LustreError: 147565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4457.987809] LustreError: 147565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4461.700428] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4463.072810] LustreError: 147566:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4466.026070] LustreError: 147594:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4466.046429] LustreError: 147594:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4466.057653] LustreError: 147594:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4468.512942] Lustre: server umount lustre-MDT0000 complete [ 4486.334846] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4498.217501] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4542.675656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4542.916993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4542.947584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4543.055706] Lustre: lustre-MDT0000: new disk, initializing [ 4543.177947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4543.210734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4547.963295] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4559.894649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4560.057846] Lustre: 150754:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4560.165729] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4560.172534] Lustre: Skipped 1 previous similar message [ 4560.301072] Lustre: lustre-MDT0001: new disk, initializing [ 4560.434072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4560.484667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4560.500090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4565.802616] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4574.411630] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4579.145590] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4586.893036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4587.166907] Lustre: lustre-OST0000: new disk, initializing [ 4587.170727] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4587.176662] Lustre: 152496:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4587.248377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4593.947420] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4594.279492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4594.292616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4594.434978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4602.471151] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4607.528741] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4609.462120] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 23:20:02 (1781234402) [ 4891.616566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4891.625158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4891.633031] Lustre: Skipped 2 previous similar messages [ 4891.653851] Lustre: Skipped 2 previous similar messages [ 4896.493119] Lustre: server umount lustre-MDT0000 complete [ 4896.735511] LustreError: 150766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4896.753749] LustreError: 150766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4900.275667] LustreError: 150746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234695 with bad export cookie 7129443604686069440 [ 4900.276184] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4900.295120] LustreError: 150746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4901.856280] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4901.860993] LustreError: 151938:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4901.867108] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4905.897956] LustreError: 153103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4905.899396] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4906.715925] Lustre: server umount lustre-MDT0001 complete [ 4917.091360] Lustre: server umount lustre-OST0000 complete [ 4928.913942] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4938.300602] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4975.160445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4975.526362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4975.557102] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4975.651824] Lustre: lustre-MDT0000: new disk, initializing [ 4975.740072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4975.767418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4980.575886] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4991.937703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4992.077264] Lustre: 157016:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4992.137380] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4992.143980] Lustre: Skipped 1 previous similar message [ 4992.242507] Lustre: lustre-MDT0001: new disk, initializing [ 4992.332363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4992.361744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4992.381171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4997.345479] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5005.523338] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5009.027682] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5017.367991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5017.631571] Lustre: lustre-OST0000: new disk, initializing [ 5017.634796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5017.640781] Lustre: 158760:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5017.722371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5024.848523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5024.882954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5025.014871] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5025.048977] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5033.665881] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5038.731240] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5039.123634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5044.710440] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5045.195196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5050.880177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5050.886984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5050.901119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5050.914412] Lustre: Skipped 1 previous similar message [ 5050.948833] Lustre: Skipped 1 previous similar message [ 5054.892633] Lustre: server umount lustre-OST0000 complete [ 5061.092111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5061.103759] Lustre: Skipped 1 previous similar message [ 5061.105255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5061.118499] Lustre: Skipped 1 previous similar message [ 5066.795318] Lustre: server umount lustre-MDT0000 complete [ 5070.305344] LustreError: 157027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5070.320155] LustreError: 157027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5070.457660] LustreError: 157008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234865 with bad export cookie 7129443604686228270 [ 5070.459936] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5070.481922] LustreError: 157008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5071.092989] Lustre: server umount lustre-MDT0001 complete [ 5080.489287] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:27:53 (1781234873) [ 5096.303876] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 5104.741313] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5144.686230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5145.007605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5145.070740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5145.175886] Lustre: lustre-MDT0000: new disk, initializing [ 5145.307528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5145.355850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5149.632652] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5160.096283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5160.246957] Lustre: 163336:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5160.547388] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5165.270427] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5173.393043] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5177.582487] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5185.053297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5185.512192] Lustre: lustre-OST0000: new disk, initializing [ 5185.516473] Lustre: Skipped 1 previous similar message [ 5185.526251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5185.535771] Lustre: Skipped 2 previous similar messages [ 5185.540852] Lustre: 165086:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5191.928502] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5193.791072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5193.805872] Lustre: Skipped 1 previous similar message [ 5193.812239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5193.897433] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5198.992435] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5209.572703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5209.585838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5209.643412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5209.653903] LustreError: Skipped 1 previous similar message [ 5209.679138] Lustre: Skipped 3 previous similar messages [ 5220.320677] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5220.511905] Lustre: server umount lustre-OST0000 complete [ 5224.931326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5224.939489] Lustre: Skipped 2 previous similar messages [ 5230.053589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5230.064787] Lustre: Skipped 7 previous similar messages [ 5230.604958] Lustre: server umount lustre-MDT0000 complete [ 5234.772418] LustreError: 163329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235029 with bad export cookie 7129443604686229152 [ 5234.812904] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5235.234283] Lustre: server umount lustre-MDT0001 complete [ 5244.665947] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:30:37 (1781235037) [ 5260.347827] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 5270.165378] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5307.644181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5307.883424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5307.908711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5307.966597] Lustre: lustre-MDT0000: new disk, initializing [ 5308.059456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5308.067688] Lustre: Skipped 2 previous similar messages [ 5308.085145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5313.011837] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5322.860285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5322.968432] Lustre: 169265:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5323.246934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5327.938510] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5336.383427] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5341.115944] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5348.295345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5348.569779] Lustre: 171017:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5353.983855] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5355.154505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5360.987503] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5370.336299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5370.345079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5370.364770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5374.029144] Lustre: server umount lustre-OST0000 complete [ 5379.559667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5379.569355] LustreError: Skipped 1 previous similar message [ 5383.721627] Lustre: server umount lustre-MDT0000 complete [ 5386.210481] LustreError: 169277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5386.229758] LustreError: 169277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5387.236585] LustreError: 169255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235181 with bad export cookie 7129443604686230237 [ 5387.253624] LustreError: 169255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5387.253642] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5387.596601] Lustre: server umount lustre-MDT0001 complete [ 5396.812096] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 5399.278456] Key type lgssc unregistered [ 5399.537630] LNet: 172740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5399.546612] LNetError: 172740:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5399.568259] LNet: Removed LNI 192.168.203.155@tcp [ 5400.289654] Key type .llcrypt unregistered [ 5400.293328] Key type ._llcrypt unregistered [ 5420.178668] Key type ._llcrypt registered [ 5420.181375] Key type .llcrypt registered [ 5420.286582] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5421.267210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5421.290085] alg: No test for adler32 (adler32-zlib) [ 5422.384656] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 5422.569137] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 5424.279223] Key type lgssc registered [ 5425.241180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5431.623330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5431.647635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5433.140492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5436.780758] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5442.530485] LustreError: 174121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5442.548465] LustreError: 174121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5444.737355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5445.069412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5449.254963] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5454.983321] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5458.477940] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5465.068657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5465.530219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5470.292595] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5475.526828] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5477.820411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5498.338703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5498.339599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5498.360304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5503.464711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5503.471316] Lustre: Skipped 2 previous similar messages [ 5504.268321] Lustre: server umount lustre-OST0000 complete [ 5508.577206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5508.590677] Lustre: Skipped 1 previous similar message [ 5508.598781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5513.698955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5513.713793] Lustre: Skipped 2 previous similar messages [ 5514.778980] Lustre: server umount lustre-MDT0000 complete [ 5518.822025] LustreError: 174120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5518.854148] LustreError: 174120:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5518.939314] LustreError: 174100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235313 with bad export cookie 15118091791057443219 [ 5518.947773] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5518.954495] LustreError: 174100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5519.298542] Lustre: server umount lustre-MDT0001 complete [ 5532.181398] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 5534.543577] Key type lgssc unregistered [ 5534.900984] LNet: 177328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5534.916601] LNetError: 177328:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5535.985547] LNet: Removed LNI 192.168.203.155@tcp [ 5536.897481] Key type .llcrypt unregistered [ 5536.899170] Key type ._llcrypt unregistered [ 5561.314731] Key type ._llcrypt registered [ 5561.317163] Key type .llcrypt registered [ 5561.425314] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5562.274848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5562.347292] alg: No test for adler32 (adler32-zlib) [ 5563.413548] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 5563.678706] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 5565.327178] Key type lgssc registered [ 5566.407357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5573.753652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5573.772913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5575.394727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5579.677385] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5585.889982] LustreError: 178731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5585.910344] LustreError: 178731:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5587.494697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5587.974477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5591.984096] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5597.549958] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5601.054280] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5608.200136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5608.533659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5614.341409] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5619.479492] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5620.846289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5626.343680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5626.345591] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5626.362361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5626.375649] Lustre: Skipped 1 previous similar message [ 5627.423897] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 5631.466088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5631.480715] Lustre: Skipped 1 previous similar message [ 5632.404653] Lustre: server umount lustre-OST0000 complete [ 5637.981763] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff97ef900bf000.ost_server_uuid 50 [ 5639.590618] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5654.502585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5654.515970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5654.529469] Lustre: Skipped 1 previous similar message [ 5654.537979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5656.613887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5659.664510] Lustre: server umount lustre-MDT0000 complete [ 5661.664657] LustreError: 178732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5661.680390] LustreError: 178732:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5663.253372] LustreError: 178711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235457 with bad export cookie 1354530208649164373 [ 5663.261510] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5663.263431] LustreError: 178711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5663.566365] Lustre: server umount lustre-MDT0001 complete [ 5672.379221] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:37:45 (1781235465) [ 5678.313381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5678.804686] LustreError: 182211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5678.889708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5683.294351] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5684.193670] LustreError: 182212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5688.298183] LustreError: 182211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5692.391288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5692.806969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5697.301091] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5703.330449] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5708.020677] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5714.368222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5714.724260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5720.173893] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5724.027737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5725.463360] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5732.557461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5732.889952] Lustre: lustre-OST0001: new disk, initializing [ 5732.899805] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5732.915333] Lustre: 184713:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5733.004937] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5739.158162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5739.171385] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5739.290102] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5739.292802] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5747.473814] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5753.400403] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 5753.405902] Lustre: Skipped 2 previous similar messages [ 5753.829589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5753.840118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5756.371321] Lustre: server umount lustre-OST0000 complete [ 5758.489871] LustreError: 183807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5763.383185] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5763.619412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5768.059552] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5768.378485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5775.336553] LustreError: 185057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5775.357501] LustreError: 185057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5785.569122] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5785.570586] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5785.581613] LustreError: Skipped 2 previous similar messages [ 5785.582293] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5785.600581] Lustre: Skipped 2 previous similar messages [ 5785.621560] Lustre: Skipped 4 previous similar messages [ 5787.531658] Lustre: server umount lustre-OST0001 complete [ 5795.296973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5795.314231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5797.441837] Lustre: server umount lustre-MDT0000 complete [ 5801.592719] LustreError: 182193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235596 with bad export cookie 1354530208649165444 [ 5801.594819] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5801.605247] LustreError: 182193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5801.982216] Lustre: server umount lustre-MDT0001 complete [ 5828.907457] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:40:21 (1781235621) [ 5837.111695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5837.464977] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5837.529596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5837.685730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5842.280563] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5852.251786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5852.369437] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5852.409641] Lustre: 188009:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5857.588631] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5863.295156] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5866.943945] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5874.029079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5874.203369] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5880.077436] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5880.475921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5886.351766] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5894.742654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5894.787908] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5902.600299] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5905.043150] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5908.710147] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5911.523882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5911.532273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5911.558250] Lustre: Skipped 1 previous similar message [ 5911.569192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5911.581295] Lustre: Skipped 2 previous similar messages [ 5917.573783] Lustre: server umount lustre-OST0000 complete [ 5919.759469] LustreError: 191010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5919.791250] LustreError: 191010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5935.587803] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5940.107870] Lustre: server umount lustre-OST0001 complete [ 5944.808210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5944.824120] Lustre: Skipped 3 previous similar messages [ 5944.831922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5944.841045] Lustre: Skipped 6 previous similar messages [ 5958.111150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5958.408600] Lustre: server umount lustre-MDT0000 complete [ 5962.499732] LustreError: 187999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235757 with bad export cookie 1354530208649166851 [ 5962.505062] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5962.516709] LustreError: 187999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5962.944975] Lustre: server umount lustre-MDT0001 complete [ 5985.727570] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:42:58 (1781235778) [ 6004.005733] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 6014.696283] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6051.481411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6051.709592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6051.740756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6051.870314] Lustre: lustre-MDT0000: new disk, initializing [ 6051.984964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6051.990988] Lustre: Skipped 3 previous similar messages [ 6052.014065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6056.405221] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6067.242335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6067.341761] Lustre: 195621:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6067.391257] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6067.395230] Lustre: Skipped 1 previous similar message [ 6067.448402] Lustre: lustre-MDT0001: new disk, initializing [ 6067.548231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6067.571499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6072.521219] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6080.703563] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6084.187811] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6090.514066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6090.701235] Lustre: lustre-OST0000: new disk, initializing [ 6090.705219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6090.712271] Lustre: 197364:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6096.080637] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6096.461260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6096.467933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6096.543905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6104.303775] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6109.790907] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6110.103798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6115.066780] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6115.283653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6117.352971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6117.356396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6117.357144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6117.357149] Lustre: Skipped 6 previous similar messages [ 6117.359150] LustreError: Skipped 3 previous similar messages [ 6117.394581] Lustre: Skipped 2 previous similar messages [ 6123.497847] Lustre: server umount lustre-OST0000 complete [ 6141.407128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6141.537732] Lustre: server umount lustre-MDT0000 complete [ 6142.945746] LustreError: 197019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6142.960995] LustreError: 197019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 6144.421947] LustreError: 195613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235939 with bad export cookie 1354530208649168636 [ 6144.422927] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6144.430888] LustreError: 195613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6144.693426] Lustre: server umount lustre-MDT0001 complete [ 6151.893640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6156.102627] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6164.012083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6167.853297] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6173.925917] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6178.317913] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6186.301477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6186.782665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6186.791938] Lustre: Skipped 4 previous similar messages [ 6193.726400] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6201.782769] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6206.333576] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6206.549575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6211.765102] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6212.108759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6214.625514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6220.143985] Lustre: server umount lustre-OST0000 complete [ 6226.606985] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6231.096822] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6260.273717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6265.944775] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6268.771994] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.776819] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6271.341876] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6278.668944] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 6284.196027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6284.197576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6284.201023] Lustre: Skipped 12 previous similar messages [ 6284.214140] Lustre: Skipped 4 previous similar messages [ 6284.288676] Lustre: server umount lustre-OST0000 complete [ 6302.175367] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6302.506356] Lustre: server umount lustre-MDT0000 complete [ 6304.739659] LustreError: 201145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6304.753217] LustreError: 201145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6306.489055] LustreError: 203135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236101 with bad export cookie 1354530208649169518 [ 6306.493821] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6306.503493] LustreError: 203135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6306.810392] Lustre: server umount lustre-MDT0001 complete [ 6316.001319] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:48:29 (1781236109) [ 6322.521514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6327.912219] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6336.814140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6341.624660] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6348.138507] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6351.899077] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6359.119154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6366.243976] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6372.158579] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6377.021379] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6377.343723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6382.809830] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6383.123375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6390.292477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6390.440679] Lustre: lustre-OST0001: new disk, initializing [ 6390.444897] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6390.451489] Lustre: 207875:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6396.861725] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6398.592218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6398.606900] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6398.729423] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6404.897417] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6409.748724] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6410.004324] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6414.358430] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6414.634819] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6419.430630] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6419.444484] LustreError: Skipped 1 previous similar message [ 6422.923230] Lustre: server umount lustre-OST0001 complete [ 6430.272098] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6430.555453] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6435.595195] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6435.883118] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6465.560515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6465.917498] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6465.928936] Lustre: Skipped 5 previous similar messages [ 6471.165982] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6471.178102] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6471.193637] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6471.217619] Lustre: Skipped 1 previous similar message [ 6472.095400] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6478.795762] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6480.312697] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6496.224757] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6496.309103] Lustre: server umount lustre-OST0001 complete [ 6519.001766] Lustre: server umount lustre-OST0000 complete [ 6537.184238] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6537.423919] Lustre: server umount lustre-MDT0000 complete [ 6541.622374] LustreError: 204861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236336 with bad export cookie 1354530208649170477 [ 6541.632512] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6541.639766] LustreError: 204861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6548.228605] Lustre: server umount lustre-MDT0001 complete [ 6575.647110] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:52:48 (1781236368) [ 6585.190889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6585.445412] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6585.469574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6591.132782] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6601.412023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6601.560430] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6601.599276] Lustre: 212830:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6607.404787] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6614.418193] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6618.861708] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6626.577848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6626.943069] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6635.121651] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6641.504648] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6651.522841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6651.668857] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6659.228618] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6666.232740] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6667.819374] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6673.002459] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6673.271986] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6678.295201] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6678.525725] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6682.694909] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97ef854c4000.ost_server_uuid 50 [ 6684.416481] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff97ef854c4000.ost_server_uuid in IDLE state after 0 sec [ 6685.554979] Lustre: Permanently deactivating lustre-OST0001 [ 6685.582941] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6689.107131] Lustre: Permanently reactivating lustre-OST0001 [ 6695.360675] Lustre: server umount lustre-OST0001 complete [ 6695.393763] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6695.413479] LustreError: Skipped 6 previous similar messages [ 6695.422729] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6695.443400] Lustre: Skipped 10 previous similar messages [ 6695.458222] LustreError: 214449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6695.480285] LustreError: 214449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 40 previous similar messages [ 6700.524702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6700.539223] Lustre: Skipped 26 previous similar messages [ 6705.658448] Lustre: server umount lustre-OST0000 complete [ 6715.366590] Lustre: server umount lustre-MDT0000 complete [ 6718.920828] LustreError: 212820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236513 with bad export cookie 1354530208649171765 [ 6718.925292] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6718.930469] LustreError: 212820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6719.329996] Lustre: server umount lustre-MDT0001 complete [ 6743.904445] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:55:37 (1781236537) [ 6752.424335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6752.822415] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6752.868778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6752.878284] Lustre: Skipped 5 previous similar messages [ 6758.480152] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6768.177869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6768.270796] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6768.291613] Lustre: 218943:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6773.387521] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6780.198374] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6784.770813] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6791.974446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6792.337911] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6792.372239] Lustre: Permanently deactivating lustre-OST0000 [ 6792.386319] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6798.906762] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6799.190986] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6804.938277] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6812.615571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6812.709235] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6819.736742] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6822.516140] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6826.006515] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6827.739152] Lustre: Permanently reactivating lustre-OST0000 [ 6827.758595] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6827.767378] Lustre: Skipped 2 previous similar messages [ 6828.212666] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6828.306241] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6828.322971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6828.343458] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6828.353694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6850.661526] Lustre: server umount lustre-OST0001 complete [ 6861.116431] Lustre: server umount lustre-OST0000 complete [ 6871.672639] Lustre: server umount lustre-MDT0000 complete [ 6875.359145] LustreError: 218936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236670 with bad export cookie 1354530208649173410 [ 6875.363596] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6875.371915] LustreError: 218936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6875.655741] Lustre: server umount lustre-MDT0001 complete [ 6887.569515] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6891.015928] Key type lgssc unregistered [ 6891.419714] LNet: 223360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6891.440080] LNetError: 223360:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6891.464248] LNet: Removed LNI 192.168.203.155@tcp [ 6892.438551] Key type .llcrypt unregistered [ 6892.443568] Key type ._llcrypt unregistered [ 6918.479724] Key type ._llcrypt registered [ 6918.482680] Key type .llcrypt registered [ 6918.576812] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6919.892328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6919.988988] alg: No test for adler32 (adler32-zlib) [ 6921.051688] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 6921.304366] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 6922.999304] Key type lgssc registered [ 6924.005154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6934.489770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6934.515824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6936.068473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6940.144261] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6946.280468] LustreError: 224810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6946.297296] LustreError: 224810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6949.700477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6949.882419] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6949.934674] Lustre: 224800:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6949.956127] Lustre: Permanently deactivating lustre-MDT0001 [ 6949.979047] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6950.377486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6951.113640] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6951.123254] LustreError: 224841:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6951.132757] LustreError: 224841:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6951.142545] LustreError: 224841:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 6955.790180] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6962.618044] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6966.919421] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6976.541808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6977.215317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6981.295851] LustreError: 226426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6981.335871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6985.484981] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6986.731752] LustreError: 226426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6991.842284] LustreError: 226427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6992.820039] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6994.931974] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7000.039190] LustreError: 226615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7000.048856] LustreError: 226615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7000.456936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7000.747582] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7006.208620] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7006.257265] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7007.613404] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7014.507392] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7018.465644] Lustre: Permanently reactivating lustre-MDT0001 [ 7018.496115] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7018.511325] Lustre: Skipped 2 previous similar messages [ 7019.926428] Lustre: lustre-MDT0001: Client b77406fd-aadc-483f-a3e1-4db7a86b5e37 (at 192.168.203.55@tcp) reconnecting [ 7025.237241] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7027.302332] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7033.883120] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7034.174479] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7039.512982] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7039.805185] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7042.550137] Lustre: Permanently deactivating lustre-MDT0001 [ 7042.564144] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7042.569255] Lustre: Skipped 2 previous similar messages [ 7049.820324] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7057.376877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7057.382837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7057.411427] Lustre: Skipped 3 previous similar messages [ 7057.443095] Lustre: Skipped 3 previous similar messages [ 7062.473859] Lustre: server umount lustre-MDT0000 complete [ 7062.509512] LustreError: 224810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7062.530672] LustreError: 224810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7067.103601] LustreError: 227906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236861 with bad export cookie 3079485544149271601 [ 7067.121915] LustreError: 227906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7067.125743] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7067.638199] Lustre: server umount lustre-MDT0001 complete [ 7074.164913] Lustre: server umount lustre-OST0000 complete [ 7086.705284] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 00:01:19 (1781236879) [ 7099.639811] Lustre: server umount lustre-OST0001 complete [ 7109.346997] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7124.025431] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7169.187144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7169.489750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7169.499876] Lustre: Skipped 2 previous similar messages [ 7169.557419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7169.674474] Lustre: lustre-MDT0000: new disk, initializing [ 7169.778932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7169.801403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7175.129119] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7186.622947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7186.804067] Lustre: 232493:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7186.885082] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7186.895798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7186.900422] Lustre: Skipped 1 previous similar message [ 7187.023175] Lustre: lustre-MDT0001: new disk, initializing [ 7187.126866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7187.191229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7187.202819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7192.020840] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7200.795674] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7204.834339] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7212.008688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7212.364391] Lustre: lustre-OST0000: new disk, initializing [ 7212.370208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7212.385856] Lustre: 234232:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7212.534602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7219.484468] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7220.815723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7220.842582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7220.952997] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7227.636800] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7230.329901] LustreError: 232498:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7239.450837] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7239.627273] Lustre: lustre-OST0001: new disk, initializing [ 7239.632078] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7239.641987] Lustre: 235235:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7240.364873] LustreError: 232498:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7240.398646] LustreError: 232499:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7246.603043] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7247.479106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7247.486910] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7247.630143] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7250.495884] LustreError: 232499:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7254.389654] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7256.124474] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7258.660702] Lustre: lustre-OST0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 7263.204554] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7263.205654] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7263.218561] LustreError: Skipped 1 previous similar message [ 7263.222517] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7263.247547] Lustre: Skipped 1 previous similar message [ 7263.268135] Lustre: Skipped 1 previous similar message [ 7264.422190] Lustre: server umount lustre-OST0001 complete [ 7268.325988] LustreError: 234215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7268.355706] LustreError: 234215:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7282.660885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7282.668986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7282.670033] LustreError: Skipped 1 previous similar message [ 7282.683562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7282.694148] Lustre: Skipped 1 previous similar message [ 7287.847212] Lustre: server umount lustre-OST0000 complete [ 7293.925624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7293.929427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7293.938563] Lustre: Skipped 2 previous similar messages [ 7293.947642] Lustre: Skipped 3 previous similar messages [ 7306.207502] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7306.535859] Lustre: server umount lustre-MDT0000 complete [ 7309.288650] LustreError: 232503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7309.301551] LustreError: 232503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 7310.432477] LustreError: 234243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237105 with bad export cookie 3079485544149274415 [ 7310.441849] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7310.444195] LustreError: 234243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7317.072760] Lustre: server umount lustre-MDT0001 complete [ 7329.268631] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7332.824700] Key type lgssc unregistered [ 7333.367827] LNet: 237142:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7333.385637] LNetError: 237142:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7333.424971] LNet: Removed LNI 192.168.203.155@tcp [ 7334.771766] Key type .llcrypt unregistered [ 7334.774426] Key type ._llcrypt unregistered [ 7375.109822] Key type ._llcrypt registered [ 7375.114635] Key type .llcrypt registered [ 7375.258480] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7376.120298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7376.216043] alg: No test for adler32 (adler32-zlib) [ 7377.248297] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 7377.409458] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7379.111357] Key type lgssc registered [ 7380.439465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7388.872880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7388.900541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.322502] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7390.345579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7390.561252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7395.904505] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7406.442148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7406.568178] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7406.634768] Lustre: 239334:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7407.018977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7412.294899] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7418.689245] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7423.075860] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7431.052203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7431.410846] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7431.686258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7438.202082] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7441.112932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7444.026682] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7497.696180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7497.701148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7497.701631] LustreError: Skipped 1 previous similar message [ 7497.703209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7497.721814] Lustre: Skipped 1 previous similar message [ 7497.736084] Lustre: Skipped 1 previous similar message [ 7502.825473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7502.835788] Lustre: Skipped 1 previous similar message [ 7503.431590] Lustre: server umount lustre-OST0000 complete [ 7508.056429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7520.422107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7520.795289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7527.003688] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7527.983432] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7528.000476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7528.007897] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7532.900973] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7573.987688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7573.992100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7573.998608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7574.019079] Lustre: Skipped 1 previous similar message [ 7578.767710] Lustre: server umount lustre-OST0000 complete [ 7584.233682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7584.260118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7584.264065] Lustre: Skipped 2 previous similar messages [ 7589.318884] Lustre: server umount lustre-MDT0000 complete [ 7594.122841] LustreError: 239327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237388 with bad export cookie 1367911936756064520 [ 7594.126241] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7594.136508] LustreError: 239327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7594.472958] Lustre: server umount lustre-MDT0001 complete [ 7605.568648] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7608.508630] Key type lgssc unregistered [ 7608.827698] LNet: 244292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7608.838178] LNetError: 244292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7608.848660] LNet: Removed LNI 192.168.203.155@tcp [ 7609.613289] Key type .llcrypt unregistered [ 7609.615591] Key type ._llcrypt unregistered [ 7630.960380] Key type ._llcrypt registered [ 7630.964141] Key type .llcrypt registered [ 7631.047806] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7632.124471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7632.198595] alg: No test for adler32 (adler32-zlib) [ 7633.426991] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 7633.695327] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7635.431200] Key type lgssc registered [ 7636.788100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7644.446880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7644.474179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7646.014391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7649.944610] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7655.400101] LustreError: 245695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7655.433443] LustreError: 245695:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7658.964224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.413378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7664.104078] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7669.882178] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7673.988669] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7681.271521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7681.743855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7688.680167] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7690.994799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7694.978583] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7720.949133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7720.959142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7720.980076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7721.959555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7721.962348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7721.963120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7722.008681] Lustre: Skipped 1 previous similar message [ 7725.975791] Lustre: server umount lustre-OST0000 complete [ 7731.168425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7731.171784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7731.197675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7736.584374] Lustre: server umount lustre-MDT0000 complete [ 7737.314457] LustreError: 245695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7737.338315] LustreError: 245695:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7740.638362] LustreError: 245676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237535 with bad export cookie 3572248021870084790 [ 7740.644476] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7740.665535] LustreError: 245676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7740.963744] Lustre: server umount lustre-MDT0001 complete [ 7751.584427] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7754.729107] Key type lgssc unregistered [ 7755.057836] LNet: 249644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7755.072249] LNetError: 249644:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7756.136931] LNet: Removed LNI 192.168.203.155@tcp [ 7757.316550] Key type .llcrypt unregistered [ 7757.318784] Key type ._llcrypt unregistered [ 7777.376949] Key type ._llcrypt registered [ 7777.379779] Key type .llcrypt registered [ 7777.510404] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7779.052695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7779.194307] alg: No test for adler32 (adler32-zlib) [ 7780.211824] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 7780.455291] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7782.119158] Key type lgssc registered [ 7783.161560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7791.490751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7791.521549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7792.949468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7796.583562] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7802.342897] LustreError: 250851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7802.363779] LustreError: 250851:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7805.636325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7806.125227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7811.080688] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7817.178345] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7821.160579] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7828.570744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7828.930821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7835.235095] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7838.202180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7841.294571] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7853.556086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7853.570449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7853.583709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7853.603349] Lustre: Skipped 1 previous similar message [ 7854.833478] Lustre: server umount lustre-OST0000 complete [ 7858.656887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7858.662774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7858.676583] Lustre: Skipped 2 previous similar messages [ 7858.700268] Lustre: Skipped 1 previous similar message [ 7863.777651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7863.785228] Lustre: Skipped 1 previous similar message [ 7864.945695] Lustre: server umount lustre-MDT0000 complete [ 7868.898917] LustreError: 250852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7868.922944] LustreError: 250852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7868.926510] LustreError: 250832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237663 with bad export cookie 5792064455230144023 [ 7868.938405] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7868.944622] LustreError: 250832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7869.280588] Lustre: server umount lustre-MDT0001 complete [ 7880.807594] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7884.024732] Key type lgssc unregistered [ 7884.324828] LNet: 254171:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7884.337291] LNetError: 254171:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7884.351892] LNet: Removed LNI 192.168.203.155@tcp [ 7885.148141] Key type .llcrypt unregistered [ 7885.150549] Key type ._llcrypt unregistered [ 7911.663907] Key type ._llcrypt registered [ 7911.667429] Key type .llcrypt registered [ 7911.781222] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7912.846735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7912.932112] alg: No test for adler32 (adler32-zlib) [ 7914.330872] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 7914.773790] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7916.495169] Key type lgssc registered [ 7917.645491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7925.337510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7925.356383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7926.862210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7931.295735] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7936.997635] LustreError: 255584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7937.023194] LustreError: 255584:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7940.550310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7940.990447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7945.635860] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7952.056519] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7956.404697] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7963.825635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7964.311356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7970.053468] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7971.505223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7976.126911] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8001.507517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8001.524831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8001.561342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8002.529443] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8002.542100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8002.543690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8002.564738] Lustre: Skipped 1 previous similar message [ 8006.351675] Lustre: server umount lustre-OST0000 complete [ 8012.769096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8012.790867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8012.809617] Lustre: Skipped 1 previous similar message [ 8016.849496] Lustre: server umount lustre-MDT0000 complete [ 8017.904767] LustreError: 255578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8017.924871] LustreError: 255578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8021.488753] LustreError: 258508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237816 with bad export cookie 14197737459883511327 [ 8021.501435] LustreError: 258508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8021.502334] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8022.032310] Lustre: server umount lustre-MDT0001 complete [ 8033.534656] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8036.361760] Key type lgssc unregistered [ 8036.773513] LNet: 259533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8036.781236] LNetError: 259533:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8037.799396] LNet: Removed LNI 192.168.203.155@tcp [ 8038.820583] Key type .llcrypt unregistered [ 8038.823094] Key type ._llcrypt unregistered [ 8058.641247] Key type ._llcrypt registered [ 8058.644405] Key type .llcrypt registered [ 8058.811487] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8059.903740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8059.919417] alg: No test for adler32 (adler32-zlib) [ 8061.237498] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 8061.579509] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8063.367217] Key type lgssc registered [ 8065.073196] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8072.575500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8072.591773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8074.089360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8078.772106] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8083.426224] LustreError: 260753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8083.439163] LustreError: 260753:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8087.386208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8087.723116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8091.211269] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8096.668568] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8100.366714] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8108.060345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8108.588900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8114.886623] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8118.846060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8120.983288] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8139.237306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8139.242245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8139.248735] LustreError: Skipped 1 previous similar message [ 8139.249546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8139.274794] Lustre: Skipped 1 previous similar message [ 8139.294299] Lustre: Skipped 1 previous similar message [ 8140.628842] Lustre: server umount lustre-OST0000 complete [ 8149.472438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8149.473055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8149.479359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8149.513474] Lustre: Skipped 1 previous similar message [ 8153.723473] Lustre: server umount lustre-MDT0000 complete [ 8154.595919] LustreError: 260747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8154.614091] LustreError: 260747:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8157.097149] LustreError: 260718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237951 with bad export cookie 4951170157200149930 [ 8157.097762] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8157.106712] LustreError: 260718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8157.430624] Lustre: server umount lustre-MDT0001 complete [ 8169.471390] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8172.419188] Key type lgssc unregistered [ 8172.661952] LNet: 264099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8172.672709] LNetError: 264099:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8172.694283] LNet: Removed LNI 192.168.203.155@tcp [ 8173.350520] Key type .llcrypt unregistered [ 8173.353286] Key type ._llcrypt unregistered [ 8184.229357] Key type ._llcrypt registered [ 8184.231471] Key type .llcrypt registered [ 8184.348308] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8201.439971] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8215.474539] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8216.536229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8216.681477] alg: No test for adler32 (adler32-zlib) [ 8217.721900] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 8217.983649] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8219.688223] Key type lgssc registered [ 8220.854133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8258.454814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8258.503541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8259.889199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8259.934849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8260.027490] Lustre: lustre-MDT0000: new disk, initializing [ 8260.164854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8260.179510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8264.806088] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8276.589534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8276.763896] Lustre: 267605:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8276.841681] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8276.841680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8276.841689] Lustre: Skipped 1 previous similar message [ 8276.938431] Lustre: lustre-MDT0001: new disk, initializing [ 8277.073203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8277.114150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8277.134695] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8281.707854] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8289.994913] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8294.194541] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8301.310209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8301.598354] Lustre: lustre-OST0000: new disk, initializing [ 8301.603776] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8301.612823] Lustre: 269345:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8301.709016] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8308.357524] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8308.364987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8308.392195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8308.523897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8316.905224] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8321.810325] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8322.050434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8326.635604] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8326.835911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8329.187091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8329.193903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8329.208797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8329.214562] Lustre: Skipped 1 previous similar message [ 8329.224605] Lustre: Skipped 1 previous similar message [ 8334.306270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8334.311045] Lustre: Skipped 1 previous similar message [ 8334.739981] Lustre: server umount lustre-OST0000 complete [ 8339.424827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8339.452715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8339.468262] Lustre: Skipped 1 previous similar message [ 8344.545798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8344.553757] Lustre: Skipped 1 previous similar message [ 8345.024358] Lustre: server umount lustre-MDT0000 complete [ 8348.571284] LustreError: 267597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238143 with bad export cookie 15391484665470917358 [ 8348.574114] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8348.595301] LustreError: 267597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8348.985881] Lustre: server umount lustre-MDT0001 complete [ 8358.962717] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 00:22:31 (1781238151) [ 8365.678508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8366.054556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8370.407529] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8371.178158] LustreError: 271610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8371.208036] LustreError: 271610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8376.297713] LustreError: 271609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8379.044946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8379.437487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8383.997198] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8389.224504] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8393.428225] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8400.388492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8400.724363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8407.082482] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8412.535705] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8422.382068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8422.386071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8422.398872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8422.406925] Lustre: Skipped 2 previous similar messages [ 8422.417832] Lustre: Skipped 1 previous similar message [ 8427.977410] Lustre: server umount lustre-OST0000 complete [ 8432.609887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8432.621739] Lustre: Skipped 1 previous similar message [ 8438.195106] Lustre: server umount lustre-MDT0000 complete [ 8442.645091] LustreError: 271590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238237 with bad export cookie 15391484665470918240 [ 8442.645505] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8442.663241] LustreError: 271590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8443.106897] Lustre: server umount lustre-MDT0001 complete [ 8456.116089] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8460.273573] Key type lgssc unregistered [ 8460.874745] LNet: 274787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8460.891891] LNetError: 274787:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8460.944781] LNet: Removed LNI 192.168.203.155@tcp [ 8462.111111] Key type .llcrypt unregistered [ 8462.113517] Key type ._llcrypt unregistered [ 8492.987973] Key type ._llcrypt registered [ 8492.994186] Key type .llcrypt registered [ 8493.179542] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8508.006673] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8509.536460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8509.597548] alg: No test for adler32 (adler32-zlib) [ 8510.892186] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 8511.264928] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8513.039176] Key type lgssc registered [ 8514.540674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8560.626925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8560.650243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8561.930417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8561.966755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8562.057561] Lustre: lustre-MDT0000: new disk, initializing [ 8562.187835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8562.209943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8566.095589] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8577.674987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8577.811953] Lustre: 278581:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8577.863213] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8577.867555] Lustre: Skipped 1 previous similar message [ 8577.954205] Lustre: lustre-MDT0001: new disk, initializing [ 8578.068889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8578.113987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8578.133709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8583.011303] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8591.040827] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8595.017283] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8602.194644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8602.498767] Lustre: lustre-OST2710: new disk, initializing [ 8602.504982] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8602.516696] Lustre: 280324:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8602.617964] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8608.309927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8608.320974] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8608.455957] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8608.538609] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8615.676661] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8622.123345] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8622.283968] Lustre: lustre-OST03e8: new disk, initializing [ 8622.287092] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8622.292317] Lustre: 281273:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8622.379584] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8628.564613] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8629.850764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8629.860472] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8629.922128] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8633.905803] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8640.999863] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8641.192674] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8644.795840] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8645.003785] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8648.906342] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8649.107684] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8652.484201] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8652.680291] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8659.940613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8659.952919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8659.967195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8660.966644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8660.977958] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8661.014924] Lustre: Skipped 2 previous similar messages [ 8662.805967] Lustre: server umount lustre-MDT0000 complete [ 8666.609983] LustreError: 278574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238461 with bad export cookie 7696940256215904094 [ 8666.613186] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8666.617333] LustreError: 278574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8666.885875] Lustre: server umount lustre-MDT0001 complete [ 8680.598797] Lustre: server umount lustre-OST2710 complete [ 8695.435053] Lustre: server umount lustre-OST03e8 complete [ 8715.896562] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8724.884994] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8762.392521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8762.843045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8762.877760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8762.979087] Lustre: lustre-MDT0000: new disk, initializing [ 8763.063557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8763.094318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8767.448739] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8779.363795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8779.607822] Lustre: 286308:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8779.697530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8779.700871] Lustre: Skipped 1 previous similar message [ 8779.871638] Lustre: lustre-MDT0001: new disk, initializing [ 8780.031209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8780.093366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8780.125216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8785.752712] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8794.075971] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8798.175889] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8805.836632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8806.143463] Lustre: lustre-OST0000: new disk, initializing [ 8806.148412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8806.154853] Lustre: 288049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8806.271961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8812.808100] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8813.105202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8813.109874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8813.248337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8819.983975] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8824.705651] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8824.896493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8829.168765] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8829.351826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8834.018226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8834.020179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8834.030192] LustreError: Skipped 1 previous similar message [ 8834.031075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8834.052437] Lustre: Skipped 1 previous similar message [ 8834.080181] Lustre: Skipped 4 previous similar messages [ 8839.139377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8839.153174] Lustre: Skipped 1 previous similar message [ 8844.259715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8845.281397] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8845.662780] Lustre: server umount lustre-OST0000 complete [ 8849.377206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8849.398034] Lustre: Skipped 1 previous similar message [ 8854.498877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8854.510139] Lustre: Skipped 4 previous similar messages [ 8855.703155] Lustre: server umount lustre-MDT0000 complete [ 8859.465251] LustreError: 286299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238654 with bad export cookie 7696940256215905774 [ 8859.476562] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8859.483564] LustreError: 286299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8859.922461] Lustre: server umount lustre-MDT0001 complete [ 8868.632691] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 00:31:01 (1781238661) [ 8870.449377] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8872.611228] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 00:31:05 (1781238665) [ 8875.940726] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8902.941497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8903.301059] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8903.335074] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8903.533675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8907.280704] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8916.251846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8916.395453] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8916.417313] Lustre: 291747:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8921.498342] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8942.514045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8942.699409] Lustre: Denying initial registration attempt from nid 192.168.203.155@tcp, specified as failover [ 8942.710648] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8942.717795] LustreError: 293166:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 8942.723857] LustreError: 293166:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 8942.729375] LustreError: 293166:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8942.777693] Lustre: server umount lustre-OST0000 complete [ 8942.780288] LustreError: 293166:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8949.728457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8949.755232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8949.763448] Lustre: Skipped 1 previous similar message [ 8955.210286] Lustre: server umount lustre-MDT0000 complete [ 8958.843285] LustreError: 291739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238753 with bad export cookie 7696940256215906656 [ 8958.869848] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8958.871749] LustreError: 291739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8959.192192] Lustre: server umount lustre-MDT0001 complete [ 8969.849614] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8972.731952] Key type lgssc unregistered [ 8972.997807] LNet: 294270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8973.002050] LNetError: 294270:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8974.053647] LNet: Removed LNI 192.168.203.155@tcp [ 8974.998885] Key type .llcrypt unregistered [ 8975.007221] Key type ._llcrypt unregistered [ 8985.891830] Key type ._llcrypt registered [ 8985.898130] Key type .llcrypt registered [ 8986.070136] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8987.134297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8987.269159] alg: No test for adler32 (adler32-zlib) [ 8988.412420] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 8988.681970] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8990.335194] Key type lgssc registered [ 8991.564300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9036.225583] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9044.832259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9044.870316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9046.188867] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9046.217493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9046.438060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9050.861695] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9058.773529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9058.850844] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9058.889542] Lustre: 297001:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9059.183835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9063.015901] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9068.176311] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9071.579599] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9092.082970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9092.328486] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9092.608989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9099.509391] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9106.381235] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9112.037357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9112.049197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9112.067779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9112.564391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9112.573987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9112.574980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9114.867603] Lustre: server umount lustre-OST0000 complete [ 9120.737059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9120.758186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9120.782869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9122.788286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9122.789672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9124.581613] Lustre: server umount lustre-MDT0000 complete [ 9129.134872] LustreError: 297724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238923 with bad export cookie 6100618110793495305 [ 9129.141669] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9129.149988] LustreError: 297724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9129.871623] Lustre: server umount lustre-MDT0001 complete [ 9141.182678] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9144.197979] Key type lgssc unregistered [ 9144.491091] LNet: 300214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9144.501303] LNetError: 300214:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9145.578506] LNet: Removed LNI 192.168.203.155@tcp [ 9146.421342] Key type .llcrypt unregistered [ 9146.423398] Key type ._llcrypt unregistered [ 9171.057470] Key type ._llcrypt registered [ 9171.059620] Key type .llcrypt registered [ 9171.179741] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9171.918562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9171.972683] alg: No test for adler32 (adler32-zlib) [ 9173.089303] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 9173.280797] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9174.935468] Key type lgssc registered [ 9176.215679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9184.610349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9184.659845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9186.144194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9190.466346] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9195.489727] LustreError: 301616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9195.508163] LustreError: 301616:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9197.149744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9197.497387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9201.727553] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9206.876746] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9210.451617] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9219.124919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9219.437645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9225.069199] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9230.310764] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9241.058048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9241.074624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9241.075760] Lustre: Skipped 1 previous similar message [ 9241.100507] Lustre: Skipped 2 previous similar messages [ 9243.182904] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 9246.073427] Lustre: server umount lustre-MDT0000 complete [ 9246.182607] LustreError: 301612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9246.208720] LustreError: 301612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9248.263371] LustreError: 303678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9249.787570] LustreError: 301597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239044 with bad export cookie 2538846964671896036 [ 9249.789745] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9249.793782] LustreError: 301597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9251.299226] LustreError: 301612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9251.300644] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9251.320963] Lustre: Skipped 1 previous similar message [ 9251.326275] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9253.416575] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 9256.354437] Lustre: server umount lustre-MDT0001 complete [ 9263.413922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9272.251828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9274.463408] LustreError: 304855:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9274.473128] LustreError: 304855:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000002a9b7b48 x1867764432138752/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1781239069 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9274.497789] LustreError: 304855:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 9274.859382] LustreError: 300924:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000064ee10a1 x1867764432139136/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9275.272733] LustreError: 303223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9275.294674] LustreError: 303223:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9275.374638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9275.468961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9280.233469] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9284.581137] LustreError: 303224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9284.595640] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9284.609903] LustreError: 303224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9288.814591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9289.233522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9293.642560] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9294.320786] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9299.172386] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9300.653358] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9304.567652] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9306.229925] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9311.203111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9311.208844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9311.231519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9311.244812] Lustre: Skipped 1 previous similar message [ 9314.789464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9314.805283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9316.752493] Lustre: server umount lustre-OST0000 complete [ 9325.026476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9325.027204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9325.030793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9325.030802] Lustre: Skipped 2 previous similar messages [ 9325.055966] Lustre: Skipped 1 previous similar message [ 9334.239279] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9334.427553] Lustre: server umount lustre-MDT0000 complete [ 9335.265292] LustreError: 304873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9335.284144] LustreError: 304873:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9337.499350] LustreError: 301596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239132 with bad export cookie 2538846964671903183 [ 9337.501876] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9337.506982] LustreError: 301596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9337.783869] Lustre: server umount lustre-MDT0001 complete [ 9346.111597] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9348.779168] Key type lgssc unregistered [ 9349.051793] LNet: 307299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9349.055540] LNetError: 307299:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9349.074093] LNet: Removed LNI 192.168.203.155@tcp [ 9349.892874] Key type .llcrypt unregistered [ 9349.894981] Key type ._llcrypt unregistered [ 9371.271454] Key type ._llcrypt registered [ 9371.282047] Key type .llcrypt registered [ 9371.387504] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9372.145558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9372.228653] alg: No test for adler32 (adler32-zlib) [ 9373.311375] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 9373.503278] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9375.167169] Key type lgssc registered [ 9376.219274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9383.295196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9383.329705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9384.798363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9388.108833] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9395.170589] LustreError: 308700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9395.192483] LustreError: 308700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9396.288293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9396.856440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9401.876687] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9407.258693] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9412.734792] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9417.185455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9417.195523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9417.216109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9419.748807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9419.752468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9421.857296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9421.875706] Lustre: Skipped 1 previous similar message [ 9423.105656] Lustre: server umount lustre-MDT0000 complete [ 9426.578313] LustreError: 308682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239221 with bad export cookie 14741933276414566696 [ 9426.581069] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9426.588152] LustreError: 308682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9426.893693] Lustre: server umount lustre-MDT0001 complete [ 9442.814618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9443.327188] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9443.402173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9443.682279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9447.881067] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9456.661288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9456.802910] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9456.835857] Lustre: 311063:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9457.134119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9461.529818] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9467.529153] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9471.495502] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9481.640648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9481.886164] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9481.898838] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9481.905762] LustreError: 312722:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 9481.910984] LustreError: 312722:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 9481.916126] LustreError: 312722:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9481.968392] Lustre: server umount lustre-OST0000 complete [ 9481.970936] LustreError: 312722:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9492.457910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9492.712327] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9492.871511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9498.878435] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9500.740214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9504.346394] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9512.726052] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9512.827809] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9512.938430] Lustre: lustre-OST0001: new disk, initializing [ 9512.949438] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9512.957783] Lustre: 314152:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9513.090858] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9519.428776] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9522.810068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9522.821692] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9522.961539] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9525.094403] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9528.295370] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9528.296737] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9528.300177] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9528.315050] Lustre: Skipped 1 previous similar message [ 9528.329867] LustreError: Skipped 1 previous similar message [ 9530.409047] Lustre: server umount lustre-OST0001 complete [ 9533.433341] LustreError: 313190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9533.448793] LustreError: 313190:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9536.481721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9536.484087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9536.489918] LustreError: Skipped 1 previous similar message [ 9536.499380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9536.499388] Lustre: Skipped 1 previous similar message [ 9538.539536] LustreError: 314626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9538.559651] LustreError: 314626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9540.497865] Lustre: server umount lustre-OST0000 complete [ 9548.774377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9548.782941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9548.792320] Lustre: Skipped 2 previous similar messages [ 9548.804556] Lustre: Skipped 4 previous similar messages [ 9550.707821] Lustre: server umount lustre-MDT0000 complete [ 9553.888674] LustreError: 312194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9553.919500] LustreError: 312194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9554.404053] LustreError: 311054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239349 with bad export cookie 14741933276414567242 [ 9554.412793] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9554.422471] LustreError: 311054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9554.842885] Lustre: server umount lustre-MDT0001 complete [ 9566.388172] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9569.265249] Key type lgssc unregistered [ 9569.555766] LNet: 315979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9569.571733] LNetError: 315979:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9569.591405] LNet: Removed LNI 192.168.203.155@tcp [ 9570.349560] Key type .llcrypt unregistered [ 9570.351348] Key type ._llcrypt unregistered [ 9649.034456] Key type ._llcrypt registered [ 9649.038926] Key type .llcrypt registered [ 9649.187426] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 9665.254309] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9666.376193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9666.403282] alg: No test for adler32 (adler32-zlib) [ 9667.378899] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [ 9667.524875] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9669.151195] Key type lgssc registered [ 9670.259182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9708.170371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9708.208256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9709.521476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9709.567117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9709.660594] Lustre: lustre-MDT0000: new disk, initializing [ 9709.729351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9709.747959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9713.523153] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9723.696060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9723.936447] Lustre: 320881:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9724.057555] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9724.073589] Lustre: Skipped 1 previous similar message [ 9724.174469] Lustre: lustre-MDT0001: new disk, initializing [ 9724.287416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9724.342659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9724.369428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9728.370505] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9736.786892] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9741.047839] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9748.833126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9749.120392] Lustre: lustre-OST0000: new disk, initializing [ 9749.124193] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9749.132105] Lustre: 322623:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9749.237200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9755.229297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9755.244073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9755.368664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9755.572951] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9763.084806] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9767.572624] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9767.826467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9772.362182] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9772.559611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9776.096271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9776.099370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9776.105478] LustreError: Skipped 1 previous similar message [ 9776.119504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9776.127615] Lustre: Skipped 1 previous similar message [ 9780.659093] Lustre: server umount lustre-OST0000 complete [ 9785.826343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9785.834515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9785.847719] Lustre: Skipped 1 previous similar message [ 9785.853740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9790.851548] Lustre: server umount lustre-MDT0000 complete [ 9794.545112] LustreError: 320873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239589 with bad export cookie 17053696954719097496 [ 9794.554266] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9794.564110] LustreError: 320873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9795.021928] Lustre: server umount lustre-MDT0001 complete [ 9803.126274] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:46:36 (1781239596) [ 9811.143413] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:46:44 (1781239604) [ 9820.265383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9820.718601] LustreError: 325231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9820.742878] LustreError: 325231:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9820.799810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9824.921085] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9826.283641] LustreError: 325232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9830.371506] LustreError: 325231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9835.253721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9835.517072] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9835.526780] Lustre: Skipped 2 previous similar messages [ 9835.883821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9841.414882] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9848.887290] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9852.573312] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9859.634275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9859.981789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9866.632841] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9873.353372] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9882.230752] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9884.336316] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9886.291661] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9888.655934] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9890.835429] Lustre: Failing over lustre-MDT0000 [ 9891.174922] Lustre: server umount lustre-MDT0000 complete [ 9891.379777] LustreError: 325227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9891.819803] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9891.836854] Lustre: Skipped 2 previous similar messages [ 9896.453531] LustreError: 325227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9896.477890] LustreError: 325227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9906.690059] LustreError: 325227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9906.700715] LustreError: 325227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9908.191537] Lustre: 319102:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781239686/real 1781239686] req@00000000aa2fc3b0 x1867764950173056/t0(0) o400->MGC192.168.203.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781239702 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9908.239720] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9911.527167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9917.758935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9917.816468] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9922.051568] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9922.921711] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9923.066156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9923.117406] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9923.173941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9930.975376] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9932.665795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9936.474372] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9938.362445] Lustre: Failing over lustre-MDT0000 [ 9938.403619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9938.410781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9938.430094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9938.436418] Lustre: Skipped 3 previous similar messages [ 9938.458352] Lustre: Skipped 2 previous similar messages [ 9938.646990] Lustre: server umount lustre-MDT0000 complete [ 9942.576241] LustreError: 325227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9942.607531] LustreError: 325227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 9958.548562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9958.711859] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9958.892456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 9958.901831] Lustre: Skipped 2 previous similar messages [ 9959.047401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9959.108543] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9964.043623] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9964.437438] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9964.547582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9964.559342] Lustre: Skipped 2 previous similar messages [ 9964.586238] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9964.678041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9973.954774] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9976.129765] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9984.996604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9984.999573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9985.015164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9985.026099] Lustre: Skipped 2 previous similar messages [ 9989.388898] Lustre: server umount lustre-MDT0000 complete [ 9990.121857] LustreError: 326900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9990.141511] LustreError: 326900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 9993.528848] LustreError: 325213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239788 with bad export cookie 17053696954719100205 [ 9993.549401] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9993.906522] Lustre: server umount lustre-MDT0001 complete [10009.188122] Lustre: server umount lustre-OST0000 complete [10026.392565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10027.100821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10032.251067] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10042.286952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10047.807644] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10053.815632] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10059.367328] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10068.039746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10071.620390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10075.977524] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10083.012704] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10095.103030] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [10095.118329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10095.137676] Lustre: Skipped 2 previous similar messages [10095.141274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10100.037444] Lustre: server umount lustre-OST0000 complete [10111.043503] Lustre: server umount lustre-MDT0000 complete [10113.505718] LustreError: 331967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10113.537683] LustreError: 331967:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10115.178713] LustreError: 332626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239909 with bad export cookie 17053696954719100954 [10115.182025] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10115.188688] LustreError: 332626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10115.633631] Lustre: server umount lustre-MDT0001 complete [10128.326176] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10131.458637] Key type lgssc unregistered [10131.762470] LNet: 334187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10131.769941] LNetError: 334187:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10131.805773] LNet: Removed LNI 192.168.203.155@tcp [10132.953548] Key type .llcrypt unregistered [10132.957157] Key type ._llcrypt unregistered [10162.967414] Key type ._llcrypt registered [10162.970150] Key type .llcrypt registered [10163.177942] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10164.293529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10164.347687] alg: No test for adler32 (adler32-zlib) [10165.450253] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [10165.707261] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10167.351176] Key type lgssc registered [10168.583133] Lustre: Echo OBD driver; http://www.lustre.org/ [10176.843686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10176.857736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10178.559447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10182.581401] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10188.772447] LustreError: 335687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10188.799034] LustreError: 335687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10192.805865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10193.075651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10197.592706] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10204.072643] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10208.540380] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10214.582144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10214.884108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10221.936580] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10224.133232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10229.101165] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10239.456984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10239.460362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10239.476809] Lustre: Skipped 2 previous similar messages [10239.504115] Lustre: Skipped 2 previous similar messages [10241.788736] Lustre: server umount lustre-MDT0000 complete [10244.586882] LustreError: 335682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10244.602174] LustreError: 335682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10246.270214] LustreError: 335666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240040 with bad export cookie 9188751673378189674 [10246.276511] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10246.286291] LustreError: 335666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10246.669984] Lustre: server umount lustre-MDT0001 complete [10261.378147] Lustre: server umount lustre-OST0000 complete [10276.303397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10276.916872] LustreError: 339058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10276.998232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10281.475869] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10282.467033] LustreError: 339059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10286.585981] LustreError: 339058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10291.258777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10291.630868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10295.910361] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10301.398527] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10305.627734] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10312.546523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10312.886131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10319.550417] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10322.174609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10325.182622] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10342.886315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10342.890653] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10342.898175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10342.934334] Lustre: Skipped 2 previous similar messages [10344.215583] Lustre: server umount lustre-MDT0000 complete [10348.004115] LustreError: 339769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10348.025712] LustreError: 339769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10348.547236] LustreError: 339039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240143 with bad export cookie 9188751673378190794 [10348.573541] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10348.910261] Lustre: server umount lustre-MDT0001 complete [10362.665798] Lustre: server umount lustre-OST0000 complete [10379.393323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10379.866158] LustreError: 342630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10379.923514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10384.364760] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10393.345890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10398.046024] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10404.350201] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10409.202217] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10416.278410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10416.780108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10416.789364] Lustre: Skipped 1 previous similar message [10423.116751] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10424.971763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10429.749854] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10435.553761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10435.555597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10435.575739] Lustre: Skipped 1 previous similar message [10435.598996] Lustre: Skipped 3 previous similar messages [10440.674114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10441.033679] Lustre: server umount lustre-OST0000 complete [10445.795465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10445.804146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10445.815484] Lustre: Skipped 1 previous similar message [10445.836111] Lustre: Skipped 2 previous similar messages [10451.640738] Lustre: server umount lustre-MDT0000 complete [10456.038149] LustreError: 342613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240250 with bad export cookie 9188751673378191816 [10456.043384] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10456.068728] LustreError: 342613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10456.540413] Lustre: server umount lustre-MDT0001 complete [10468.602288] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10471.588945] Key type lgssc unregistered [10471.946064] LNet: 345808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10471.960702] LNetError: 345808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10471.991446] LNet: Removed LNI 192.168.203.155@tcp [10472.978630] Key type .llcrypt unregistered [10472.984805] Key type ._llcrypt unregistered [10501.906929] Key type ._llcrypt registered [10501.910193] Key type .llcrypt registered [10502.031851] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10503.076332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10503.279067] alg: No test for adler32 (adler32-zlib) [10504.294541] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [10504.565822] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10506.200910] Key type lgssc registered [10507.470209] Lustre: Echo OBD driver; http://www.lustre.org/ [10515.299165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10515.327302] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10515.340212] LustreError: 347234:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10515.372567] LustreError: 347234:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10515.384279] LustreError: 347234:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10515.394446] LustreError: 347234:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10515.415900] LustreError: 347234:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10524.475113] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10524.481208] LustreError: 347631:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10524.505657] LustreError: 347631:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10524.514242] LustreError: 347631:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10524.523600] LustreError: 347631:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10524.530811] LustreError: 347631:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10546.869500] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10549.981168] Key type lgssc unregistered [10550.316265] LNet: 348618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10550.324507] LNetError: 348618:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10550.351634] LNet: Removed LNI 192.168.203.155@tcp [10551.359100] Key type .llcrypt unregistered [10551.364403] Key type ._llcrypt unregistered [10571.185920] Key type ._llcrypt registered [10571.189712] Key type .llcrypt registered [10571.311277] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [10587.477303] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10588.580687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10588.744711] alg: No test for adler32 (adler32-zlib) [10590.008111] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [10590.189785] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10591.863164] Key type lgssc registered [10593.219548] Lustre: Echo OBD driver; http://www.lustre.org/ [10630.602979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10630.619756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10631.974570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10632.018983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10632.114651] Lustre: lustre-MDT0000: new disk, initializing [10632.235938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10632.265254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10636.799980] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10648.491684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10648.678563] Lustre: 351742:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10648.780572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10648.789385] Lustre: Skipped 1 previous similar message [10648.958571] Lustre: lustre-MDT0001: new disk, initializing [10649.117714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10649.180540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10649.194240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10653.706556] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10661.754083] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10666.231733] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10674.360366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10674.693870] Lustre: lustre-OST0000: new disk, initializing [10674.697863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10674.702873] Lustre: 353498:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10674.791742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10681.627903] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10682.990736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10683.006315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10683.177897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10689.806615] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10694.446684] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10694.737754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10699.853887] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10700.149705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10703.841973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10703.844747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10703.849033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10703.860606] Lustre: Skipped 1 previous similar message [10703.871171] Lustre: Skipped 1 previous similar message [10708.656776] Lustre: server umount lustre-OST0000 complete [10713.058464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10713.066819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10718.177956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10718.186602] Lustre: Skipped 1 previous similar message [10723.309669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10723.320154] Lustre: Skipped 1 previous similar message [10727.392591] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10728.129441] Lustre: server umount lustre-MDT0000 complete [10732.780467] LustreError: 353497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240527 with bad export cookie 2992885357357025671 [10732.783352] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10732.791564] LustreError: 353497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10733.319753] Lustre: server umount lustre-MDT0001 complete [10743.315363] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10745.168447] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 01:02:18 (1781240538) [10752.056631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10752.350559] LustreError: 355814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10752.364734] LustreError: 355814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10752.408165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10756.139442] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10757.602199] LustreError: 355815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10762.730416] LustreError: 355814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10764.228058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10764.717567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10769.585654] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10776.348966] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10780.677112] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10788.329562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10788.817268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10795.754643] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10801.806841] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10808.967097] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10809.194554] Lustre: lustre-OST0001: new disk, initializing [10809.203296] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10809.208296] Lustre: 358312:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10809.302846] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10815.092350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10815.111499] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10815.266665] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10815.964472] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10824.954526] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10830.818322] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10830.821622] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10830.823947] LustreError: Skipped 2 previous similar messages [10830.827096] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10830.844860] Lustre: Skipped 2 previous similar messages [10836.651255] Lustre: server umount lustre-OST0001 complete [10838.566344] LustreError: 357407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10846.182779] LustreError: 357407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10846.213211] LustreError: 357407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10856.416152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10856.418279] LustreError: 357407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10856.418591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10856.430342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10856.430351] Lustre: Skipped 4 previous similar messages [10856.467664] LustreError: 357407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10869.216279] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10869.498990] Lustre: server umount lustre-OST0000 complete [10876.897667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10876.906107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10876.936017] Lustre: Skipped 2 previous similar messages [10876.968156] Lustre: Skipped 6 previous similar messages [10880.682184] Lustre: server umount lustre-MDT0000 complete [10882.018358] LustreError: 355811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10882.032205] LustreError: 355811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10885.264351] LustreError: 355796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240679 with bad export cookie 2992885357357026553 [10885.272362] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10885.288362] LustreError: 355796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10891.687885] Lustre: server umount lustre-MDT0001 complete [10903.924649] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10907.121725] Key type lgssc unregistered [10907.458687] LNet: 360171:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10907.461343] LNetError: 360171:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10907.476588] LNet: Removed LNI 192.168.203.155@tcp [10908.516356] Key type .llcrypt unregistered [10908.519113] Key type ._llcrypt unregistered [10960.413414] Key type ._llcrypt registered [10960.424593] Key type .llcrypt registered [10960.551801] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10962.112788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10962.177623] alg: No test for adler32 (adler32-zlib) [10963.349987] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [10963.578972] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10965.295244] Key type lgssc registered [10966.739246] Lustre: Echo OBD driver; http://www.lustre.org/ [10975.394652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10975.432579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10976.876666] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10976.903185] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10977.129700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10982.353554] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10992.031865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10992.173981] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10992.235397] Lustre: 362958:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10992.546331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10997.898927] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11004.649571] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11008.636501] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11013.093753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11013.106203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11013.125330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11013.607732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11013.636164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11018.720680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11018.730150] Lustre: Skipped 1 previous similar message [11023.841265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11023.849800] Lustre: Skipped 1 previous similar message [11025.375273] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11025.593154] Lustre: server umount lustre-MDT0000 complete [11029.352065] LustreError: 362951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240824 with bad export cookie 13346596825087425959 [11029.357768] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11029.368094] LustreError: 362951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11029.725410] Lustre: server umount lustre-MDT0001 complete [11039.742642] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 01:07:12 (1781240832) [11046.612041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11047.015171] LustreError: 365245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11047.038898] LustreError: 365245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11047.090566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11052.033900] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11052.517281] LustreError: 365246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11057.636725] LustreError: 365245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11061.129425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11061.439662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11066.170799] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11072.402992] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11076.687766] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11083.605697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11083.840669] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11084.048538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11090.776391] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11097.394315] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11105.197799] Lustre: Permanently deactivating lustre-OST0000 [11105.215519] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11111.952371] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11113.127783] LustreError: 367842:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11113.140343] LustreError: 367842:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11113.151846] LustreError: 367842:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11116.880286] LustreError: 367891:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11116.890407] LustreError: 367891:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11125.257776] Lustre: server umount lustre-OST0000 complete [11128.638623] LustreError: 368140:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11128.652132] LustreError: 368140:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11131.363117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11131.374044] Lustre: Skipped 1 previous similar message [11131.375751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11131.395964] Lustre: Skipped 1 previous similar message [11140.837910] Lustre: server umount lustre-MDT0000 complete [11141.601864] LustreError: 365245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11141.625070] LustreError: 365245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11145.189662] LustreError: 365226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240939 with bad export cookie 13346596825087426540 [11145.198428] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11145.199405] LustreError: 365226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11154.561816] Lustre: server umount lustre-MDT0001 complete [11163.334171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11168.690581] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11171.746264] LustreError: 369285:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11179.272626] Lustre: 369624:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.155@tcp [11181.773859] Lustre: server umount MGS complete [11195.464544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11196.042523] LustreError: 370349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11196.068977] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11196.078404] Lustre: Skipped 1 previous similar message [11196.182325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11201.436643] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11206.636670] LustreError: 370349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11206.656019] LustreError: 370349:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11210.580949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11210.900889] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11215.648771] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11222.111865] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11226.752946] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11228.965417] Lustre: Permanently reactivating lustre-OST0000 [11228.980229] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11228.990407] Lustre: Skipped 2 previous similar messages [11257.311244] LustreError: 371904:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11266.529172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11266.964602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11266.969563] Lustre: Skipped 1 previous similar message [11274.425165] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11281.715556] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11292.640543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11292.667518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11292.677266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11292.682542] Lustre: Skipped 2 previous similar messages [11293.671873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11295.112737] Lustre: server umount lustre-OST0000 complete [11303.392300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11303.398209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11303.417142] Lustre: Skipped 1 previous similar message [11306.435715] Lustre: server umount lustre-MDT0000 complete [11309.026289] LustreError: 370346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11309.050351] LustreError: 370346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11311.355600] LustreError: 370329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241106 with bad export cookie 13346596825087427590 [11311.361078] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11311.373292] LustreError: 370329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11311.818683] Lustre: server umount lustre-MDT0001 complete [11325.908417] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11329.456458] Key type lgssc unregistered [11329.846626] LNet: 374357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11329.855995] LNetError: 374357:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11329.889466] LNet: Removed LNI 192.168.203.155@tcp [11330.849322] Key type .llcrypt unregistered [11330.850970] Key type ._llcrypt unregistered [11350.786231] Key type ._llcrypt registered [11350.788623] Key type .llcrypt registered [11350.894298] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [11368.833352] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11370.329519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11370.367459] alg: No test for adler32 (adler32-zlib) [11371.616579] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [11371.967096] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11373.673061] Key type lgssc registered [11374.988436] Lustre: Echo OBD driver; http://www.lustre.org/ [11418.299251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11418.322736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11419.805972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11419.866139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11420.016503] Lustre: lustre-MDT0000: new disk, initializing [11420.232035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11420.286168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11424.484195] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11436.977300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11437.137590] Lustre: 377485:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11437.182129] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11437.186191] Lustre: Skipped 1 previous similar message [11437.245292] Lustre: lustre-MDT0001: new disk, initializing [11437.376659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11437.434876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11437.448757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11442.043329] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11450.454536] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11455.384761] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11463.615971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11463.933794] Lustre: lustre-OST0000: new disk, initializing [11463.938748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11463.945655] Lustre: 379227:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11464.049685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11469.379044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11469.395511] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11469.515866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11471.239231] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11479.975306] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11484.565171] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11484.785450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11489.888656] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11490.153497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11495.405359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11495.405438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11495.412411] LustreError: Skipped 1 previous similar message [11495.413774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11495.447581] Lustre: Skipped 1 previous similar message [11498.506875] Lustre: server umount lustre-OST0000 complete [11504.096102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11504.105857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11504.123530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11504.134269] Lustre: Skipped 1 previous similar message [11505.639775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11505.644506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11505.660064] Lustre: Skipped 1 previous similar message [11508.665369] Lustre: server umount lustre-MDT0000 complete [11512.232300] LustreError: 377476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241306 with bad export cookie 4518584135186658046 [11512.233628] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11512.245621] LustreError: 377476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11512.733865] Lustre: server umount lustre-MDT0001 complete [11522.629981] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11524.612882] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 01:15:17 (1781241317) [11531.489199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11532.041587] LustreError: 381541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11532.064000] LustreError: 381541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11532.125410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11536.183864] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11537.380244] LustreError: 381542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11541.476111] LustreError: 381541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11545.282810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11545.729228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11550.459288] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11557.170395] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11561.671689] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11569.079810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11569.540194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11576.985087] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11584.102856] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11586.252682] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11588.791680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11596.264378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11596.288672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11596.304550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11601.856932] Lustre: server umount lustre-OST0000 complete [11607.009457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11607.022931] LustreError: Skipped 1 previous similar message [11607.039888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11607.056375] Lustre: Skipped 1 previous similar message [11607.068071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11607.072762] Lustre: Skipped 2 previous similar messages [11612.882687] Lustre: server umount lustre-MDT0000 complete [11616.977473] LustreError: 381521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241411 with bad export cookie 4518584135186658928 [11616.985286] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11616.987614] LustreError: 381521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11617.412881] Lustre: server umount lustre-MDT0001 complete [11628.386537] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11630.911142] Key type lgssc unregistered [11631.194143] LNet: 384909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11631.206797] LNetError: 384909:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11632.232917] LNet: Removed LNI 192.168.203.155@tcp [11633.068451] Key type .llcrypt unregistered [11633.075311] Key type ._llcrypt unregistered [11656.433889] Key type ._llcrypt registered [11656.438675] Key type .llcrypt registered [11656.570513] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11656.928312] Key type .llcrypt unregistered [11656.930549] Key type ._llcrypt unregistered [11674.670487] Key type ._llcrypt registered [11674.674534] Key type .llcrypt registered [11674.877315] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11676.200307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11676.277770] alg: No test for adler32 (adler32-zlib) [11677.452908] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [11677.884343] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11679.623526] Key type lgssc registered [11681.171837] Lustre: Echo OBD driver; http://www.lustre.org/ [11690.675376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11690.703695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11692.394144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11697.874349] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11702.756470] LustreError: 387089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11702.790945] LustreError: 387089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11707.791968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11707.872784] LustreError: 387090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11708.214403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11713.510173] LustreError: 387089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11713.545565] LustreError: 387089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11715.829561] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11717.602463] LustreError: 387085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11717.628246] LustreError: 387085:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11717.641024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11722.391568] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11722.722520] LustreError: 387084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11722.751462] LustreError: 387084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11729.306157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11729.560339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11733.910189] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11745.259794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11745.260118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11745.271370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11745.282206] Lustre: Skipped 1 previous similar message [11749.865695] Lustre: server umount lustre-OST0000 complete [11755.488439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11755.493906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11755.518964] Lustre: Skipped 1 previous similar message [11755.522164] Lustre: Skipped 1 previous similar message [11759.786908] Lustre: server umount lustre-MDT0000 complete [11760.612319] LustreError: 387090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11760.644377] LustreError: 387090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11763.501965] LustreError: 387071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241558 with bad export cookie 622853304392730493 [11763.512575] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11763.535099] LustreError: 387071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11763.910253] Lustre: server umount lustre-MDT0001 complete [11774.382817] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11777.342801] Key type lgssc unregistered [11777.653853] LNet: 390073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11777.662944] LNetError: 390073:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11777.673755] LNet: Removed LNI 192.168.203.155@tcp [11778.555992] Key type .llcrypt unregistered [11778.564551] Key type ._llcrypt unregistered [11803.309094] Key type ._llcrypt registered [11803.311617] Key type .llcrypt registered [11803.405355] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11804.533789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11804.556307] alg: No test for adler32 (adler32-zlib) [11805.620642] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [11805.873662] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11807.511166] Key type lgssc registered [11808.728489] Lustre: Echo OBD driver; http://www.lustre.org/ [11817.323032] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11817.349278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11832.993793] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11838.111220] LustreError: 391448:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [11864.802877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11871.348508] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11877.646886] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11885.146253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11885.752778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11885.851712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11891.178791] LustreError: 391473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11891.203162] LustreError: 391473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11891.373805] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11896.288338] LustreError: 391474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11899.373988] LustreError: 391473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11899.401740] LustreError: 391473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11901.914178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11902.614249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11907.531585] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11913.419222] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11917.503633] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11927.024222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11927.039522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11927.069369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11927.972236] Lustre: server umount lustre-OST0000 complete [11932.713359] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11932.727319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11932.753532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11938.788645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11938.805688] Lustre: Skipped 3 previous similar messages [11939.022595] Lustre: server umount lustre-MDT0000 complete [11943.561765] LustreError: 391454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241738 with bad export cookie 16024048516561457669 [11943.564812] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11943.567950] LustreError: 391454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11954.347500] Lustre: server umount lustre-MDT0001 complete [11965.580211] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11968.289240] Key type lgssc unregistered [11968.607641] LNet: 394665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11968.616831] LNetError: 394665:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11968.630468] LNet: Removed LNI 192.168.203.155@tcp [11969.463169] Key type .llcrypt unregistered [11969.467275] Key type ._llcrypt unregistered [11991.998801] Key type ._llcrypt registered [11992.002488] Key type .llcrypt registered [11992.143178] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11993.285789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11993.423791] alg: No test for adler32 (adler32-zlib) [11994.437263] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [11994.715728] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11996.498183] Key type lgssc registered [11997.704825] Lustre: Echo OBD driver; http://www.lustre.org/ [12005.404488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12005.463820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12007.036572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12010.817897] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12016.616360] LustreError: 396065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12016.634941] LustreError: 396065:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12019.441920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12019.931431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12024.315160] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12030.736759] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12035.137176] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12042.392750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12042.876455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12049.074952] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12052.094705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12054.633287] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12061.153471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12061.158650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12061.173348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12062.238833] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [12062.690257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12062.702145] Lustre: Skipped 1 previous similar message [12065.341436] Lustre: server umount lustre-MDT0000 complete [12067.414304] LustreError: 396047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.55@tcp arrived at 1781241862 with bad export cookie 2702214015522369411 [12067.442707] LustreError: 396047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12067.809107] LustreError: 397176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12067.828738] LustreError: 397176:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12069.413492] LustreError: 396045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241864 with bad export cookie 2702214015522368515 [12069.418031] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12069.429663] LustreError: 396045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12070.881452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12070.888343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12070.918213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12070.929925] Lustre: Skipped 3 previous similar messages [12072.933329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12072.937727] LustreError: 396066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12072.950858] LustreError: 396066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12074.976425] LustreError: 397176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12075.007782] LustreError: 397176:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12075.662589] Lustre: server umount lustre-OST0000 complete [12080.097951] LustreError: 396066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12080.118782] LustreError: 396066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12083.776369] Lustre: server umount lustre-MDT0001 complete [12095.005087] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12098.092848] Key type lgssc unregistered [12098.448902] LNet: 399395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12098.456534] LNetError: 399395:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12098.473977] LNet: Removed LNI 192.168.203.155@tcp [12099.410142] Key type .llcrypt unregistered [12099.412093] Key type ._llcrypt unregistered [12122.641315] Key type ._llcrypt registered [12122.645589] Key type .llcrypt registered [12122.748283] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12123.719083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12123.878738] alg: No test for adler32 (adler32-zlib) [12124.892929] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [12125.236689] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12126.919179] Key type lgssc registered [12128.033833] Lustre: Echo OBD driver; http://www.lustre.org/ [12135.073633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12135.089339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12136.521080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12140.739993] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12146.657031] LustreError: 400795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12146.676329] LustreError: 400795:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12149.452741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12149.793567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12154.658423] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12160.231394] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12164.074401] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12171.294756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12171.595850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12178.408880] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12181.873627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12185.865635] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12190.697078] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12190.716888] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12190.752669] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12191.978655] Lustre: server umount lustre-MDT0001 complete [12192.232937] LustreError: 401506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12192.239418] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12197.349033] LustreError: 400796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12197.367084] LustreError: 400796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12202.467394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12202.470655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12202.476133] LustreError: 400795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12202.482698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12205.815845] Lustre: server umount lustre-OST0000 complete [12207.597368] LustreError: 400796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12209.837642] Lustre: server umount lustre-MDT0000 complete [12222.040452] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12224.482451] Key type lgssc unregistered [12224.764360] LNet: 404121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12224.772921] LNetError: 404121:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12224.786429] LNet: Removed LNI 192.168.203.155@tcp [12225.508672] Key type .llcrypt unregistered [12225.510368] Key type ._llcrypt unregistered [12250.064376] Key type ._llcrypt registered [12250.066631] Key type .llcrypt registered [12250.215717] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [12265.451371] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12266.670151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12266.735525] alg: No test for adler32 (adler32-zlib) [12267.864804] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [12268.134103] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12269.839189] Key type lgssc registered [12271.009911] Lustre: Echo OBD driver; http://www.lustre.org/ [12313.169761] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12330.815574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12330.916117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12332.239469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12332.309780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12332.458862] Lustre: lustre-MDT0000: new disk, initializing [12332.586170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12332.608644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12337.484659] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12352.931278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12353.152522] Lustre: lustre-OST0000: new disk, initializing [12353.155875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12353.161421] Lustre: Skipped 1 previous similar message [12353.166546] Lustre: 409092:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12353.224898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12358.796561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12358.812499] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12358.819559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12358.847390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12366.363168] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12383.273375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12383.464500] Lustre: 408087:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12383.597497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12383.777690] Lustre: lustre-MDT0001: new disk, initializing [12383.906947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12383.945706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12383.953061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12389.076929] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12399.037431] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12399.285446] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12404.209679] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12404.407642] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12409.906624] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [12413.922758] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12413.956194] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12413.964213] Lustre: Skipped 1 previous similar message [12414.975516] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [12415.589843] Lustre: server umount lustre-MDT0001 complete [12424.170934] LustreError: 408094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12424.196294] LustreError: 408094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12429.283639] LustreError: 408094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12429.317411] LustreError: 408094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12434.400522] LustreError: 408099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12434.407178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12434.442222] Lustre: Skipped 1 previous similar message [12434.450382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12439.523279] LustreError: 408100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12439.547634] LustreError: 408100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12439.735636] Lustre: server umount lustre-OST0000 complete [12449.823164] Lustre: server umount lustre-MDT0000 complete [12465.812437] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12469.168676] Key type lgssc unregistered [12469.608130] LNet: 412655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12469.634050] LNetError: 412655:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12469.697166] LNet: Removed LNI 192.168.203.155@tcp [12470.441798] Key type .llcrypt unregistered [12470.446176] Key type ._llcrypt unregistered [12517.627310] Key type ._llcrypt registered [12517.633108] Key type .llcrypt registered [12517.772387] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12518.095325] Key type .llcrypt unregistered [12518.098740] Key type ._llcrypt unregistered [12535.151436] Key type ._llcrypt registered [12535.154520] Key type .llcrypt registered [12535.280703] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12536.530947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12536.617197] alg: No test for adler32 (adler32-zlib) [12537.692828] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [12537.903844] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12539.631489] Key type lgssc registered [12541.115643] Lustre: Echo OBD driver; http://www.lustre.org/ [12575.135565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12575.188352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12576.411849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12576.439640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12576.580795] Lustre: lustre-MDT0000: new disk, initializing [12576.705622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12576.731881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12581.344124] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12592.990079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12593.122435] Lustre: 416002:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12593.172971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12593.178371] Lustre: Skipped 1 previous similar message [12593.276936] Lustre: lustre-MDT0001: new disk, initializing [12593.371050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12593.400966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12593.408463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12598.411376] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12607.058329] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12611.252787] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12618.733630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12618.980950] Lustre: lustre-OST0000: new disk, initializing [12618.985483] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12618.990326] Lustre: 417745:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12619.080382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12624.463371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12624.475561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12624.626321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12625.022667] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12632.964037] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12639.714092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12639.730515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12639.759977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12640.236081] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12645.355898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12645.374557] Lustre: Skipped 5 previous similar messages [12645.745909] Lustre: server umount lustre-MDT0000 complete [12649.507372] LustreError: 415996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242444 with bad export cookie 2407602206895726105 [12649.508686] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12649.520224] LustreError: 415996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12650.046724] Lustre: server umount lustre-MDT0001 complete [12664.319509] Lustre: server umount lustre-OST0000 complete [12675.781208] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 01:34:28 (1781242468) [12684.140582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12684.821581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12689.641780] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12689.889845] LustreError: 419660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12689.917548] LustreError: 419660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12695.016152] LustreError: 419659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12699.141979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12699.489991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12703.756498] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12709.929820] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12713.969717] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12734.005568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12734.320254] Lustre: Found index 0 for lustre-OST0000, updating log [12734.546285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12740.394781] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12746.269825] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12756.099883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12766.691559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12766.693452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12766.695300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12766.698635] LustreError: Skipped 1 previous similar message [12766.724212] Lustre: Skipped 2 previous similar messages [12768.431678] Lustre: server umount lustre-OST0000 complete [12784.096995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12784.099557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12784.116265] Lustre: Skipped 1 previous similar message [12784.137991] Lustre: Skipped 4 previous similar messages [12789.218103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12790.610579] Lustre: server umount lustre-MDT0000 complete [12793.316940] LustreError: 419659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12793.331981] LustreError: 419659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12794.975948] LustreError: 419639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242589 with bad export cookie 2407602206895727498 [12794.987027] LustreError: 419639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12794.993049] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12795.660405] Lustre: server umount lustre-MDT0001 complete [12801.060644] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp [12804.262476] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 01:36:37 (1781242597) [13421.147452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13421.599021] LustreError: 425755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13421.627070] LustreError: 425755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13421.667410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13425.905910] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13427.170874] LustreError: 425756:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13431.271484] LustreError: 425755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13433.744257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13434.101630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13438.281481] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13443.529673] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13447.088554] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13465.645647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13465.801084] Lustre: Found index 0 for lustre-OST0000, updating log [13466.086486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13470.189896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13471.433483] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13476.422225] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13479.967322] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp [13480.417529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13480.428510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13480.441858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13480.448639] Lustre: Skipped 1 previous similar message [13492.191080] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13492.384838] Lustre: server umount lustre-OST0000 complete [13495.776694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13495.784580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13495.796711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13496.807423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13497.891369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13497.905795] Lustre: Skipped 3 previous similar messages [13501.865932] Lustre: server umount lustre-MDT0000 complete [13501.991949] LustreError: 425756:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13502.017760] LustreError: 425756:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13504.865386] LustreError: 425737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243299 with bad export cookie 2407602206895728583 [13504.875708] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13504.878719] LustreError: 425737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13505.199780] Lustre: server umount lustre-MDT0001 complete [13531.503408] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [13538.666493] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13543.457493] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp [13568.574938] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:49:21 (1781243361) [13583.473302] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [13585.872251] Key type lgssc unregistered [13586.104792] LNet: 431975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13586.109391] LNetError: 431975:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13682.408711] LNet: Removed LNI 192.168.203.155@tcp [13683.050255] Key type .llcrypt unregistered [13683.052183] Key type ._llcrypt unregistered [13693.827574] Key type ._llcrypt registered [13693.829924] Key type .llcrypt registered [13693.898950] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13694.471168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13694.489369] alg: No test for adler32 (adler32-zlib) [13695.461693] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [13695.609915] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [13697.239153] Key type lgssc registered [13698.010522] Lustre: Echo OBD driver; http://www.lustre.org/ [13701.096232] LNet: Added LNI 192.168.203.155@tcp100 [8/256/0/180] [13701.715909] LNet: Added LNI 192.168.203.155@tcp101 [8/256/0/180] [13702.282273] LNet: Added LNI 192.168.203.155@tcp102 [8/256/0/180] [13702.908762] LNet: Added LNI 192.168.203.155@tcp103 [8/256/0/180] [13703.587724] LNet: Added LNI 192.168.203.155@tcp104 [8/256/0/180] [13704.161845] LNet: Added LNI 192.168.203.155@tcp105 [8/256/0/180] [13704.736537] LNet: Added LNI 192.168.203.155@tcp106 [8/256/0/180] [13705.274699] LNet: Added LNI 192.168.203.155@tcp107 [8/256/0/180] [13705.855427] LNet: Added LNI 192.168.203.155@tcp108 [8/256/0/180] [13706.437492] LNet: Added LNI 192.168.203.155@tcp109 [8/256/0/180] [13707.017302] LNet: Added LNI 192.168.203.155@tcp110 [8/256/0/180] [13707.548036] LNet: Added LNI 192.168.203.155@tcp111 [8/256/0/180] [13708.152665] LNet: Added LNI 192.168.203.155@tcp112 [8/256/0/180] [13708.772227] LNet: Added LNI 192.168.203.155@tcp113 [8/256/0/180] [13709.375625] LNet: Added LNI 192.168.203.155@tcp114 [8/256/0/180] [13709.959387] LNet: Added LNI 192.168.203.155@tcp115 [8/256/0/180] [13710.498956] LNet: Added LNI 192.168.203.155@tcp116 [8/256/0/180] [13711.101442] LNet: Added LNI 192.168.203.155@tcp117 [8/256/0/180] [13711.755483] LNet: Added LNI 192.168.203.155@tcp118 [8/256/0/180] [13712.264375] LNet: Added LNI 192.168.203.155@tcp119 [8/256/0/180] [13712.889920] LNet: Added LNI 192.168.203.155@tcp120 [8/256/0/180] [13713.573410] LNet: Added LNI 192.168.203.155@tcp121 [8/256/0/180] [13714.243594] LNet: Added LNI 192.168.203.155@tcp122 [8/256/0/180] [13714.963765] LNet: Added LNI 192.168.203.155@tcp123 [8/256/0/180] [13715.570756] LNet: Added LNI 192.168.203.155@tcp124 [8/256/0/180] [13716.154027] LNet: Added LNI 192.168.203.155@tcp125 [8/256/0/180] [13716.745543] LNet: Added LNI 192.168.203.155@tcp126 [8/256/0/180] [13717.412635] LNet: Added LNI 192.168.203.155@tcp127 [8/256/0/180] [13718.041124] LNet: Added LNI 192.168.203.155@tcp128 [8/256/0/180] [13718.626596] LNet: Added LNI 192.168.203.155@tcp129 [8/256/0/180] [13719.257086] LNet: Added LNI 192.168.203.155@tcp130 [8/256/0/180] [13719.888755] LNet: Added LNI 192.168.203.155@tcp131 [8/256/0/180] [13720.499458] LNet: Added LNI 192.168.203.155@tcp132 [8/256/0/180] [13721.038110] LNet: Added LNI 192.168.203.155@tcp133 [8/256/0/180] [13721.592481] LNet: Added LNI 192.168.203.155@tcp134 [8/256/0/180] [13722.245145] LNet: Added LNI 192.168.203.155@tcp135 [8/256/0/180] [13726.542494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13726.581944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13727.681680] Lustre: 434966:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [13727.730924] LNetError: 432426:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13727.743356] LNetError: Protocol error connecting to 192.168.203.155@tcp119 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13727.748539] LNetError: Skipped 2 previous similar messages [13727.834128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13727.853897] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13727.919258] Lustre: lustre-MDT0000: new disk, initializing [13727.995386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13728.016598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13728.416621] LNet: 7 local NIs in recovery (showing 7): 192.168.203.155@tcp125, 192.168.203.155@tcp124, 192.168.203.155@tcp127, 192.168.203.155@tcp119, 192.168.203.155@tcp126, 192.168.203.155@tcp128, 192.168.203.155@tcp129 [13728.418155] LNetError: 434989:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13728.439495] LNetError: 434989:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [13728.458147] LNetError: Protocol error connecting to 192.168.203.155@tcp115 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13728.470933] LNetError: Skipped 5 previous similar messages [13730.650666] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13737.481935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13737.576988] Lustre: 434983:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13737.623629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13737.628523] Lustre: Skipped 1 previous similar message [13737.678156] Lustre: lustre-MDT0001: new disk, initializing [13737.704873] Lustre: 435767:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [13737.713373] Lustre: 435767:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13737.784574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13737.810797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13737.820704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13740.625719] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13742.637608] Lustre: 436115:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [13742.645058] Lustre: 436115:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13746.177878] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13748.713284] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13753.110395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13753.309311] Lustre: lustre-OST0000: new disk, initializing [13753.313082] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13753.318478] Lustre: 436731:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13753.383445] Lustre: 436736:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [13753.391949] Lustre: 436736:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [13753.430579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13757.506165] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13758.116480] LNetError: 432427:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13758.127672] LNetError: Protocol error connecting to 192.168.203.155@tcp120 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13760.161350] LNetError: 432424:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13760.167755] LNetError: Protocol error connecting to 192.168.203.155@tcp123 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13761.551730] Lustre: 437282:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [13761.557143] Lustre: 437282:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13761.592246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13761.599232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13761.676717] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13762.589534] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13771.424624] LNetError: 432426:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13771.434815] LNetError: 432426:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [13771.441480] LNetError: Protocol error connecting to 192.168.203.155@tcp120 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13771.449891] LNetError: Skipped 1 previous similar message [13772.268710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13772.270264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13772.272936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13772.285309] Lustre: Skipped 1 previous similar message [13772.290929] LustreError: Skipped 1 previous similar message [13773.649738] Lustre: server umount lustre-OST0000 complete [13777.378884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13777.387448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13777.391697] Lustre: Skipped 1 previous similar message [13782.498373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13782.502798] Lustre: Skipped 2 previous similar messages [13784.737975] LNetError: 434990:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13784.748362] LNetError: 434990:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [13784.753969] LNetError: Protocol error connecting to 192.168.203.155@tcp120 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13784.760841] LNetError: Skipped 2 previous similar messages [13787.616633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13791.199115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13791.318885] Lustre: server umount lustre-MDT0000 complete [13793.929788] LustreError: 434974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243588 with bad export cookie 15495166744165927685 [13793.931640] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13793.943291] LustreError: 434974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13794.161250] Lustre: server umount lustre-MDT0001 complete [13801.204075] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [13802.145952] LNetError: 432425:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [13802.152743] LNetError: 432425:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [13802.158319] LNetError: Protocol error connecting to 192.168.203.155@tcp121 at host 192.168.203.155:988: is it running a compatible version of Lustre? [13802.165897] LNetError: Skipped 4 previous similar messages [13803.080208] Key type lgssc unregistered [13803.369288] LNet: 438513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13803.378029] LNetError: 438513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13803.382974] LNet: Removed LNI 192.168.203.155@tcp [13804.447153] LNet: Removed LNI 192.168.203.155@tcp100 [13804.449843] LNetError: 438513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13804.453685] LNetError: 438513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13804.457575] LNet: Removed LNI 192.168.203.155@tcp101 [13805.472527] LNet: Removed LNI 192.168.203.155@tcp102 [13805.475588] LNetError: 438513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13805.481978] LNetError: 438513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13805.487659] LNet: Removed LNI 192.168.203.155@tcp103 [13806.496654] LNet: Removed LNI 192.168.203.155@tcp104 [13807.519145] LNet: Removed LNI 192.168.203.155@tcp105 [13807.520980] LNetError: 438513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13807.528464] LNetError: 438513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13807.534995] LNet: Removed LNI 192.168.203.155@tcp106 [13808.543140] LNet: Removed LNI 192.168.203.155@tcp107 [13809.567656] LNet: Removed LNI 192.168.203.155@tcp108 [13810.591208] LNet: Removed LNI 192.168.203.155@tcp109 [13811.615358] LNet: Removed LNI 192.168.203.155@tcp110 [13811.617666] LNetError: 438513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13811.622445] LNetError: 438513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13811.625987] LNet: Removed LNI 192.168.203.155@tcp111 [13812.639365] LNet: Removed LNI 192.168.203.155@tcp112 [13813.663140] LNet: Removed LNI 192.168.203.155@tcp113 [13814.687127] LNet: Removed LNI 192.168.203.155@tcp114 [13815.711200] LNet: Removed LNI 192.168.203.155@tcp115 [13816.735643] LNet: Removed LNI 192.168.203.155@tcp116 [13817.759138] LNet: Removed LNI 192.168.203.155@tcp117 [13818.783143] LNet: Removed LNI 192.168.203.155@tcp118 [13819.807188] LNet: Removed LNI 192.168.203.155@tcp119 [13819.812558] LNetError: 438513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13819.819719] LNetError: 438513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13819.829433] LNet: Removed LNI 192.168.203.155@tcp120 [13820.895627] LNet: Removed LNI 192.168.203.155@tcp121 [13821.919148] LNet: Removed LNI 192.168.203.155@tcp122 [13822.943264] LNet: Removed LNI 192.168.203.155@tcp123 [13823.967158] LNet: Removed LNI 192.168.203.155@tcp124 [13824.991851] LNet: Removed LNI 192.168.203.155@tcp125 [13826.015190] LNet: Removed LNI 192.168.203.155@tcp126 [13827.040706] LNet: Removed LNI 192.168.203.155@tcp127 [13828.063240] LNet: Removed LNI 192.168.203.155@tcp128 [13829.087154] LNet: Removed LNI 192.168.203.155@tcp129 [13829.094950] LNet: Removed LNI 192.168.203.155@tcp130 [13829.098930] LNet: Removed LNI 192.168.203.155@tcp131 [13829.105993] LNet: Removed LNI 192.168.203.155@tcp132 [13829.108734] LNet: Removed LNI 192.168.203.155@tcp133 [13829.113639] LNet: Removed LNI 192.168.203.155@tcp134 [13829.129877] LNet: Removed LNI 192.168.203.155@tcp135 [13829.607511] Key type .llcrypt unregistered [13829.609341] Key type ._llcrypt unregistered [13843.120986] Key type ._llcrypt registered [13843.124059] Key type .llcrypt registered [13843.183452] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [13852.242166] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13852.975177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13852.987748] alg: No test for adler32 (adler32-zlib) [13853.953484] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [13854.124182] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [13855.735274] Key type lgssc registered [13856.373589] Lustre: Echo OBD driver; http://www.lustre.org/ [13878.072578] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:54:31 (1781243671) [13888.962551] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [13890.634673] Key type lgssc unregistered [13890.830548] LNet: 442527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13890.835669] LNetError: 442527:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13890.848409] LNet: Removed LNI 192.168.203.155@tcp [13891.335670] Key type .llcrypt unregistered [13891.337511] Key type ._llcrypt unregistered [13901.853438] Key type ._llcrypt registered [13901.855426] Key type .llcrypt registered [13901.934277] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13902.346362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13902.372551] alg: No test for adler32 (adler32-zlib) [13903.308474] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [13903.447844] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [13905.071175] Key type lgssc registered [13905.782398] Lustre: Echo OBD driver; http://www.lustre.org/ [13911.483618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13911.496847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13912.643591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13912.672468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13912.731674] Lustre: lustre-MDT0000: new disk, initializing [13912.787085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13912.810429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13915.357216] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13928.693859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13928.889742] Lustre: 443701:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13929.176178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13929.182318] Lustre: Skipped 1 previous similar message [13929.240211] Lustre: lustre-MDT0001: new disk, initializing [13929.310173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13929.350254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13929.360857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13935.066639] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13946.722060] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13952.757837] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13963.498936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13964.029217] Lustre: lustre-OST0000: new disk, initializing [13964.038054] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13964.059993] Lustre: 445443:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13964.262817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13973.141905] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13974.076988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13974.091587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13974.248724] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13981.810381] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13986.937445] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing lnet_if_list [13992.295608] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13992.328945] LNet: Added LNI 192.168.203.155@tcp10 [8/256/0/180] [13998.028351] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13998.077242] LNet: Added LNI 192.168.203.155@tcp11 [8/256/0/180] [14002.821747] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14002.845461] LNet: Added LNI 192.168.203.155@tcp12 [8/256/0/180] [14008.112109] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14008.175780] LNet: Added LNI 192.168.203.155@tcp13 [8/256/0/180] [14013.667301] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14013.740263] LNet: Added LNI 192.168.203.155@tcp14 [8/256/0/180] [14018.765082] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14018.786905] LNet: Added LNI 192.168.203.155@tcp15 [8/256/0/180] [14023.124674] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14023.173938] LNet: Added LNI 192.168.203.155@tcp16 [8/256/0/180] [14027.709918] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14027.747360] LNet: Added LNI 192.168.203.155@tcp17 [8/256/0/180] [14032.286981] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14032.316994] LNet: Added LNI 192.168.203.155@tcp18 [8/256/0/180] [14037.313875] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14037.360218] LNet: Added LNI 192.168.203.155@tcp19 [8/256/0/180] [14042.453159] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14042.500979] LNet: Added LNI 192.168.203.155@tcp20 [8/256/0/180] [14047.296394] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14047.322471] LNet: Added LNI 192.168.203.155@tcp21 [8/256/0/180] [14052.430726] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14052.456577] LNet: Added LNI 192.168.203.155@tcp22 [8/256/0/180] [14057.695207] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14057.726230] LNet: Added LNI 192.168.203.155@tcp23 [8/256/0/180] [14063.093794] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14063.124935] LNet: Added LNI 192.168.203.155@tcp24 [8/256/0/180] [14068.677699] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14068.741722] LNet: Added LNI 192.168.203.155@tcp25 [8/256/0/180] [14074.341650] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14074.383434] LNet: Added LNI 192.168.203.155@tcp26 [8/256/0/180] [14080.082077] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14080.123812] LNet: Added LNI 192.168.203.155@tcp27 [8/256/0/180] [14085.852691] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14085.873748] LNet: Added LNI 192.168.203.155@tcp28 [8/256/0/180] [14091.910207] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14091.951474] LNet: Added LNI 192.168.203.155@tcp29 [8/256/0/180] [14098.044187] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14098.063446] LNet: Added LNI 192.168.203.155@tcp30 [8/256/0/180] [14103.755747] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14103.785693] LNet: Added LNI 192.168.203.155@tcp31 [8/256/0/180] [14109.803416] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14109.833454] LNet: Added LNI 192.168.203.155@tcp32 [8/256/0/180] [14115.858682] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14115.911428] LNet: Added LNI 192.168.203.155@tcp33 [8/256/0/180] [14122.196981] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14122.222481] LNet: Added LNI 192.168.203.155@tcp34 [8/256/0/180] [14128.870102] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14128.894738] LNet: Added LNI 192.168.203.155@tcp35 [8/256/0/180] [14135.724557] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14135.762649] LNet: Added LNI 192.168.203.155@tcp36 [8/256/0/180] [14143.434673] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14143.467679] LNet: Added LNI 192.168.203.155@tcp37 [8/256/0/180] [14150.391634] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14150.436739] LNet: Added LNI 192.168.203.155@tcp38 [8/256/0/180] [14157.096156] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14157.132268] LNet: Added LNI 192.168.203.155@tcp39 [8/256/0/180] [14163.657654] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14163.706189] LNet: Added LNI 192.168.203.155@tcp40 [8/256/0/180] [14170.642980] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14170.669667] LNet: Added LNI 192.168.203.155@tcp41 [8/256/0/180] [14177.746883] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14177.797329] LNet: Added LNI 192.168.203.155@tcp42 [8/256/0/180] [14185.057521] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14185.085369] LNet: Added LNI 192.168.203.155@tcp43 [8/256/0/180] [14192.861664] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14192.885067] LNet: Added LNI 192.168.203.155@tcp44 [8/256/0/180] [14200.786983] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14200.831033] LNet: Added LNI 192.168.203.155@tcp45 [8/256/0/180] [14209.357568] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14209.433517] LNet: Added LNI 192.168.203.155@tcp46 [8/256/0/180] [14217.472611] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14217.507538] LNet: Added LNI 192.168.203.155@tcp47 [8/256/0/180] [14226.085651] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14226.113501] LNet: Added LNI 192.168.203.155@tcp48 [8/256/0/180] [14235.878489] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14235.915010] LNet: Added LNI 192.168.203.155@tcp49 [8/256/0/180] [14246.117274] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14246.156930] LNet: Added LNI 192.168.203.155@tcp50 [8/256/0/180] [14267.368658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14267.378797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14267.400107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14271.463284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14271.465887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14271.484902] Lustre: Skipped 1 previous similar message [14271.504567] Lustre: Skipped 2 previous similar messages [14273.521833] Lustre: server umount lustre-MDT0000 complete [14277.759158] LustreError: 443692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244072 with bad export cookie 8008755770045233722 [14277.760516] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14277.776548] LustreError: 443692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14278.224767] Lustre: server umount lustre-MDT0001 complete [14298.079672] Lustre: 442975:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781244076/real 1781244076] req@00000000b1342491 x1867769392044672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781244092 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14298.124689] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14299.103118] Lustre: 442974:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781244077/real 1781244077] req@0000000044258c21 x1867769392044800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781244093 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14300.252946] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14303.199157] Lustre: 442972:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781244081/real 1781244081] req@00000000bf5cad30 x1867769392044928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781244097 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14308.321098] Lustre: 442974:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781244087/real 1781244087] req@00000000afd1d19d x1867769392045312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781244103 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14308.351287] Lustre: 442974:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14309.538635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14313.503150] Lustre: 442974:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781244092/real 1781244092] req@000000003bcc2e60 x1867769392045440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781244108 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14322.723860] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6f24d0f0ce680482 [14322.740980] Lustre: MGC192.168.203.155@tcp: Connection restored to 0@lo (at 0@lo) [14323.106951] LustreError: 445437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14323.134574] LustreError: 445437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14323.186709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14327.207365] LustreError: 445438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14328.117254] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14328.241740] LustreError: 445437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14328.251285] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14333.416396] LustreError: 445438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14337.508279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14337.966302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14343.086073] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14347.196498] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14353.719432] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14359.106675] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14384.101263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14384.103140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14384.116988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14384.128796] Lustre: Skipped 1 previous similar message [14387.722094] Lustre: server umount lustre-OST0000 complete [14394.336635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14394.339619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14394.357991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14394.371616] Lustre: Skipped 2 previous similar messages [14394.390429] Lustre: Skipped 1 previous similar message [14397.503603] Lustre: server umount lustre-MDT0000 complete [14399.456031] LustreError: 468860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14399.489887] LustreError: 468860:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14401.243192] LustreError: 443693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244195 with bad export cookie 8008755770045236354 [14401.246846] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14401.253369] LustreError: 443693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14401.621615] Lustre: server umount lustre-MDT0001 complete [14412.558752] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [14415.469829] Key type lgssc unregistered [14415.885064] LNet: 471416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14415.891944] LNetError: 471416:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14415.901888] LNet: Removed LNI 192.168.203.155@tcp [14415.907995] LNet: Removed LNI 192.168.203.155@tcp10 [14415.911345] LNet: Removed LNI 192.168.203.155@tcp11 [14415.920447] LNet: Removed LNI 192.168.203.155@tcp12 [14415.923868] LNet: Removed LNI 192.168.203.155@tcp13 [14415.929429] LNet: Removed LNI 192.168.203.155@tcp14 [14415.932274] LNet: Removed LNI 192.168.203.155@tcp15 [14415.937172] LNet: Removed LNI 192.168.203.155@tcp16 [14415.939217] LNet: Removed LNI 192.168.203.155@tcp17 [14415.942425] LNet: Removed LNI 192.168.203.155@tcp18 [14415.945618] LNet: Removed LNI 192.168.203.155@tcp19 [14415.948253] LNet: Removed LNI 192.168.203.155@tcp20 [14415.951097] LNet: Removed LNI 192.168.203.155@tcp21 [14415.953928] LNet: Removed LNI 192.168.203.155@tcp22 [14415.956263] LNet: Removed LNI 192.168.203.155@tcp23 [14415.961543] LNet: Removed LNI 192.168.203.155@tcp24 [14415.965336] LNet: Removed LNI 192.168.203.155@tcp25 [14415.968279] LNet: Removed LNI 192.168.203.155@tcp26 [14415.971817] LNet: Removed LNI 192.168.203.155@tcp27 [14415.973871] LNet: Removed LNI 192.168.203.155@tcp28 [14415.976305] LNet: Removed LNI 192.168.203.155@tcp29 [14415.978356] LNet: Removed LNI 192.168.203.155@tcp30 [14415.980437] LNet: Removed LNI 192.168.203.155@tcp31 [14415.983214] LNet: Removed LNI 192.168.203.155@tcp32 [14415.985782] LNet: Removed LNI 192.168.203.155@tcp33 [14415.988227] LNet: Removed LNI 192.168.203.155@tcp34 [14415.990579] LNet: Removed LNI 192.168.203.155@tcp35 [14415.993235] LNet: Removed LNI 192.168.203.155@tcp36 [14415.995711] LNet: Removed LNI 192.168.203.155@tcp37 [14415.998198] LNet: Removed LNI 192.168.203.155@tcp38 [14416.000914] LNet: Removed LNI 192.168.203.155@tcp39 [14416.003341] LNet: Removed LNI 192.168.203.155@tcp40 [14416.005753] LNet: Removed LNI 192.168.203.155@tcp41 [14416.008115] LNet: Removed LNI 192.168.203.155@tcp42 [14416.010744] LNet: Removed LNI 192.168.203.155@tcp43 [14416.013213] LNet: Removed LNI 192.168.203.155@tcp44 [14416.015780] LNet: Removed LNI 192.168.203.155@tcp45 [14416.018313] LNet: Removed LNI 192.168.203.155@tcp46 [14416.021165] LNet: Removed LNI 192.168.203.155@tcp47 [14416.023836] LNet: Removed LNI 192.168.203.155@tcp48 [14416.026266] LNet: Removed LNI 192.168.203.155@tcp49 [14416.036901] LNet: Removed LNI 192.168.203.155@tcp50 [14416.912656] Key type .llcrypt unregistered [14416.915159] Key type ._llcrypt unregistered [14435.591955] Key type ._llcrypt registered [14435.607327] Key type .llcrypt registered [14435.746996] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [14452.873338] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14454.067906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14454.211598] alg: No test for adler32 (adler32-zlib) [14455.334494] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [14455.634115] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [14457.335701] Key type lgssc registered [14458.716785] Lustre: Echo OBD driver; http://www.lustre.org/ [14496.840615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14496.880197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14498.219310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14498.263359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14498.377783] Lustre: lustre-MDT0000: new disk, initializing [14498.473989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14498.495522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14502.807227] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14513.930589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14513.995574] Lustre: 474582:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14514.056672] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14514.067410] Lustre: Skipped 1 previous similar message [14514.182281] Lustre: lustre-MDT0001: new disk, initializing [14514.284738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14514.338504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14514.355258] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14518.572437] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14527.583766] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14532.917953] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14541.328215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14541.527809] Lustre: *** cfs_fail_loc=90f, val=0*** [14541.597891] Lustre: lustre-OST0000: new disk, initializing [14541.601225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14541.608484] Lustre: 476372:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14541.722546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14547.813055] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14548.596748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14548.607229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14548.758872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14555.333863] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14569.443756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14569.445317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14569.455038] LustreError: Skipped 1 previous similar message [14569.465735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14569.471605] Lustre: Skipped 1 previous similar message [14569.502330] Lustre: Skipped 1 previous similar message [14570.737543] Lustre: server umount lustre-OST0000 complete [14575.584456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14575.586928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14575.605625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14579.684622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14579.686615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14584.802178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14584.808046] Lustre: Skipped 2 previous similar messages [14589.407170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14589.512900] LustreError: 477412:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000d9112d47) refcount nonzero (1) after lock cleanup; forcing cleanup. [14589.774327] Lustre: server umount lustre-MDT0000 complete [14593.528126] LustreError: 474576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244388 with bad export cookie 16256902267740455474 [14593.543060] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14593.543502] LustreError: 474576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14593.945655] Lustre: server umount lustre-MDT0001 complete [14607.284883] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [14611.435134] Key type lgssc unregistered [14611.781712] LNet: 478148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14611.790303] LNetError: 478148:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14611.822113] LNet: Removed LNI 192.168.203.155@tcp [14612.717412] Key type .llcrypt unregistered [14612.722189] Key type ._llcrypt unregistered [14634.124703] Key type ._llcrypt registered [14634.135144] Key type .llcrypt registered [14634.267826] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14635.178636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14635.317875] alg: No test for adler32 (adler32-zlib) [14636.353511] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [14636.547897] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [14638.215917] Key type lgssc registered [14639.615763] Lustre: Echo OBD driver; http://www.lustre.org/ [14647.847534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14647.881623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14649.589185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14654.111023] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14660.069283] LustreError: 479403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14660.086038] LustreError: 479403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14663.136462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14663.536739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14668.355176] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14674.233244] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14678.246419] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14685.245675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14685.600756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14691.633818] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14698.307255] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14710.530978] loop0: detected capacity change from 0 to 204800000 [14710.744923] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14710.764871] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14710.775941] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14718.107870] loop0: detected capacity change from 0 to 204800000 [14718.256151] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14718.266963] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14718.274379] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14723.328401] loop0: detected capacity change from 0 to 204800000 [14728.327296] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14728.565418] Lustre: fs15246-OST0000: new disk, initializing [14728.579878] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14728.583292] Lustre: 482653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14728.640709] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14735.021717] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14745.722093] loop1: detected capacity change from 0 to 204800000 [14750.870703] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14751.049642] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14751.122410] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14751.234559] Lustre: fs15246-MDT0000: new disk, initializing [14751.375529] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14751.435173] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14751.615482] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14752.682666] Lustre: 483759:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000052211be8 x1867770160525824/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781244563 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14756.331949] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14761.832268] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14761.840605] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14761.877474] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14786.023985] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14786.052191] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14787.662705] Lustre: server umount fs15246-MDT0000 complete [14791.163424] LustreError: 481581:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14796.264632] LustreError: 481581:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14801.046183] Lustre: server umount fs15246-OST0000 complete [14816.738263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14816.758421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14816.764212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14816.771252] Lustre: Skipped 1 previous similar message [14816.786034] Lustre: Skipped 1 previous similar message [14819.222427] Lustre: server umount lustre-OST0000 complete [14826.977303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14826.996541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14829.928312] Lustre: server umount lustre-MDT0000 complete [14832.105627] LustreError: 480113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14832.133400] LustreError: 480113:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14834.212774] LustreError: 479382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244628 with bad export cookie 5484907569526110111 [14834.227619] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14834.232957] LustreError: 479382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14834.567335] Lustre: server umount lustre-MDT0001 complete [14845.852630] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [14848.931470] Key type lgssc unregistered [14849.282109] LNet: 486825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14849.289368] LNetError: 486825:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14849.310493] LNet: Removed LNI 192.168.203.155@tcp [14850.059959] Key type .llcrypt unregistered [14850.063722] Key type ._llcrypt unregistered [14873.747603] Key type ._llcrypt registered [14873.752597] Key type .llcrypt registered [14873.887202] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14875.253190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14875.379550] alg: No test for adler32 (adler32-zlib) [14876.542951] Lustre: Lustre: Build Version: 2.17.53_61_g0d029bc [14876.836558] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [14878.576044] Key type lgssc registered [14880.010923] Lustre: Echo OBD driver; http://www.lustre.org/ [14922.698706] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [14932.506782] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14976.670587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14976.706716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14978.215783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14978.272813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14978.496057] Lustre: lustre-MDT0000: new disk, initializing [14978.711488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14978.735040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14985.261821] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14996.159833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14996.308441] Lustre: 491245:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14996.389496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14996.396888] Lustre: Skipped 1 previous similar message [14996.470854] Lustre: lustre-MDT0001: new disk, initializing [14996.610866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14996.668160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14996.682619] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15001.243632] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [15010.556453] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15014.711238] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15022.771415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15023.134685] Lustre: lustre-OST0000: new disk, initializing [15023.139785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15023.147925] Lustre: 492988:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15023.245426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15028.867185] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [15030.825199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15030.833979] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15030.972410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15036.500294] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15041.495470] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15041.688793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15046.673619] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15046.864869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15051.754372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15051.769195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15051.771265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15051.792221] Lustre: Skipped 1 previous similar message [15055.348335] Lustre: server umount lustre-OST0000 complete [15061.986025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15061.995632] Lustre: Skipped 1 previous similar message [15062.002533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15062.005759] Lustre: Skipped 1 previous similar message [15066.153888] Lustre: server umount lustre-MDT0000 complete [15069.738232] LustreError: 491236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244864 with bad export cookie 14614560175728004295 [15069.748826] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15070.118037] Lustre: server umount lustre-MDT0001 complete [15095.447250] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [15105.399692] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [15137.115430] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14897 sec ========= 02:15:30 (1781244930) [15138.564406] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:15:31 (1781244931) === [15142.124951] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:15:35 (1781244935) === [15181.906339] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [15184.680295] Key type lgssc unregistered [15184.990950] LNet: 499432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15185.002368] LNetError: 499432:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15185.021828] LNet: Removed LNI 192.168.203.155@tcp [15185.855245] Key type .llcrypt unregistered [15185.857602] Key type ._llcrypt unregistered