[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496362272 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2866568K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002305] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.006409] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009115] LSM: Security Framework initializing [ 0.010050] Yama: becoming mindful. [ 0.011038] SELinux: Initializing. [ 0.012053] *** VALIDATE selinux *** [ 0.019577] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023618] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026097] *** VALIDATE tmpfs *** [ 0.027398] *** VALIDATE proc *** [ 0.028160] *** VALIDATE cgroup *** [ 0.029007] *** VALIDATE cgroup2 *** [ 0.030196] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031101] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033026] Spectre V2 : User space: Vulnerable [ 0.034011] Speculative Store Bypass: Vulnerable [ 0.037268] debug: unmapping init [mem 0xffffffff90259000-0xffffffff90260fff] [ 0.039779] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040601] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041022] ... version: 2 [ 0.042014] ... bit width: 48 [ 0.043013] ... generic registers: 4 [ 0.044015] ... value mask: 0000ffffffffffff [ 0.045013] ... max period: 00007fffffffffff [ 0.046018] ... fixed-purpose events: 3 [ 0.047015] ... event mask: 000000070000000f [ 0.048280] rcu: Hierarchical SRCU implementation. [ 0.050498] smp: Bringing up secondary CPUs ... [ 0.051588] x86: Booting SMP configuration: [ 0.052023] .... node #0, CPUs: #1 #2 #3 [ 0.055200] smp: Brought up 1 node, 4 CPUs [ 0.057012] smpboot: Max logical packages: 1 [ 0.058017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.128925] node 0 deferred pages initialised in 67ms [ 0.132311] devtmpfs: initialized [ 0.134262] x86/mm: Memory block size: 128MB [ 0.136836] gcov: version magic: 0x41383552 [ 0.140155] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.143115] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146330] pinctrl core: initialized pinctrl subsystem [ 0.148181] [ 0.148840] ************************************************************* [ 0.151010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153011] ** ** [ 0.156012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158009] ** ** [ 0.162014] ** This means that this kernel is built to expose internal ** [ 0.164010] ** IOMMU data structures, which may compromise security on ** [ 0.167011] ** your system. ** [ 0.170012] ** ** [ 0.173013] ** If you see this message and you are not debugging the ** [ 0.176012] ** kernel, report this immediately to your vendor! ** [ 0.178010] ** ** [ 0.180012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183023] ************************************************************* [ 0.185690] NET: Registered protocol family 16 [ 0.186439] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.192076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196111] cpuidle: using governor menu [ 0.198000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201375] PCI: Using configuration type 1 for base access [ 0.203126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.211089] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.212019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.213177] cryptd: max_cpu_qlen set to 1000 [ 0.215319] ACPI: Added _OSI(Module Device) [ 0.217018] ACPI: Added _OSI(Processor Device) [ 0.218011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.220010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.224000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.232842] ACPI: Interpreter enabled [ 0.234055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.236010] ACPI: Using IOAPIC for interrupt routing [ 0.238123] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.241646] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.252465] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.255036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.258018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.261146] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266384] acpiphp: Slot [2] registered [ 0.268134] acpiphp: Slot [5] registered [ 0.269114] acpiphp: Slot [6] registered [ 0.270131] acpiphp: Slot [7] registered [ 0.272181] acpiphp: Slot [8] registered [ 0.274148] acpiphp: Slot [9] registered [ 0.277162] acpiphp: Slot [10] registered [ 0.279184] acpiphp: Slot [3] registered [ 0.281151] acpiphp: Slot [4] registered [ 0.283105] acpiphp: Slot [11] registered [ 0.286112] acpiphp: Slot [12] registered [ 0.288114] acpiphp: Slot [13] registered [ 0.290141] acpiphp: Slot [14] registered [ 0.293128] acpiphp: Slot [15] registered [ 0.295110] acpiphp: Slot [16] registered [ 0.297205] acpiphp: Slot [17] registered [ 0.299101] acpiphp: Slot [18] registered [ 0.300109] acpiphp: Slot [19] registered [ 0.301108] acpiphp: Slot [20] registered [ 0.302115] acpiphp: Slot [21] registered [ 0.303161] acpiphp: Slot [22] registered [ 0.304082] acpiphp: Slot [23] registered [ 0.305170] acpiphp: Slot [24] registered [ 0.307075] acpiphp: Slot [25] registered [ 0.309104] acpiphp: Slot [26] registered [ 0.310103] acpiphp: Slot [27] registered [ 0.312103] acpiphp: Slot [28] registered [ 0.314119] acpiphp: Slot [29] registered [ 0.315057] acpiphp: Slot [30] registered [ 0.316000] acpiphp: Slot [31] registered [ 0.316000] PCI host bridge to bus 0000:00 [ 0.318021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.321021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.323027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.326023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.330028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.334025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.336178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.339150] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.343644] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.354025] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.358542] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.362021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.364017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.368023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.372014] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.374791] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.376038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.380059] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.385016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.400017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.408021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.414680] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.429020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.440019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.478019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.489263] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.495013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.502017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.518014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.529300] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.537017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.545064] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.556017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.567650] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.574013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.579017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.595030] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.605656] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.613021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.623015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.641016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.652699] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.667013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.677016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.705017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.714625] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.717385] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.719443] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.724688] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.729237] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.734282] iommu: Default domain type: Passthrough [ 0.735000] SCSI subsystem initialized [ 0.736191] ACPI: bus type USB registered [ 0.739160] usbcore: registered new interface driver usbfs [ 0.742260] usbcore: registered new interface driver hub [ 0.745137] usbcore: registered new device driver usb [ 0.748370] pps_core: LinuxPPS API ver. 1 registered [ 0.750017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.755084] PTP clock support registered [ 0.757375] EDAC MC: Ver: 3.0.0 [ 0.760207] PCI: Using ACPI for IRQ routing [ 0.763473] NetLabel: Initializing [ 0.765018] NetLabel: domain hash size = 128 [ 0.768129] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.770141] NetLabel: unlabeled traffic allowed by default [ 0.773334] vgaarb: loaded [ 0.775069] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.777019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.783978] clocksource: Switched to clocksource kvm-clock [ 0.907213] VFS: Disk quotas dquot_6.6.0 [ 0.909077] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.912092] *** VALIDATE ramfs *** [ 0.913799] *** VALIDATE hugetlbfs *** [ 0.915533] pnp: PnP ACPI init [ 0.918340] pnp: PnP ACPI: found 6 devices [ 0.935786] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.939656] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.942095] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.944357] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.946911] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.949466] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.952570] NET: Registered protocol family 2 [ 0.955378] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.960576] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.966745] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.972114] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.975425] TCP: Hash tables configured (established 65536 bind 65536) [ 0.978709] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.982162] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.985445] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.988773] NET: Registered protocol family 1 [ 0.992375] RPC: Registered named UNIX socket transport module. [ 0.995344] RPC: Registered udp transport module. [ 0.996947] RPC: Registered tcp transport module. [ 0.999054] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.001458] NET: Registered protocol family 44 [ 1.003782] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.006238] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.008956] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.011950] PCI: CLS 0 bytes, default 64 [ 1.014115] Unpacking initramfs... [ 2.511242] debug: unmapping init [mem 0xffff8b4afcc54000-0xffff8b4afffbffff] [ 2.515888] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.518628] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.522363] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.046802] Initialise system trusted keyrings [ 3.049119] Key type blacklist registered [ 3.052436] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.065587] zbud: loaded [ 3.070138] *** VALIDATE nfs *** [ 3.071509] *** VALIDATE nfs4 *** [ 3.073517] pstore: using deflate compression [ 3.078623] Platform Keyring initialized [ 3.191502] NET: Registered protocol family 38 [ 3.193686] Key type asymmetric registered [ 3.195276] Asymmetric key parser 'x509' registered [ 3.197135] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.200737] io scheduler mq-deadline registered [ 3.202596] io scheduler kyber registered [ 3.203843] io scheduler bfq registered [ 3.205350] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.208529] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.211512] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.214709] ACPI: Power Button [PWRF] [ 3.220066] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.226760] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.243779] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.251329] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.274289] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.302272] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.335673] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.340386] Non-volatile memory driver v1.3 [ 3.342125] Linux agpgart interface v0.103 [ 3.374291] virtio_blk virtio1: [vda] 146672 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.377379] vda: detected capacity change from 0 to 75096064 [ 3.393646] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.397505] vdb: detected capacity change from 0 to 1073741824 [ 3.422334] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.425644] vdc: detected capacity change from 0 to 2621440000 [ 3.449065] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.452606] vdd: detected capacity change from 0 to 2621440000 [ 3.469039] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.472590] vde: detected capacity change from 0 to 4294967296 [ 3.487587] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.490637] vdf: detected capacity change from 0 to 4294967296 [ 3.498206] libphy: Fixed MDIO Bus: probed [ 3.507934] usbcore: registered new interface driver usbserial_generic [ 3.510129] usbserial: USB Serial support registered for generic [ 3.512740] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.517243] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.519975] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.522646] mousedev: PS/2 mouse device common for all mice [ 3.525972] rtc_cmos 00:05: RTC can wake from S4 [ 3.529375] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.531077] rtc_cmos 00:05: registered as rtc0 [ 3.536883] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.536924] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.542406] intel_pstate: CPU model not supported [ 3.546040] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.551116] hid: raw HID events driver (C) Jiri Kosina [ 3.554137] usbcore: registered new interface driver usbhid [ 3.557226] usbhid: USB HID core driver [ 3.559953] drop_monitor: Initializing network drop monitor service [ 3.563050] Initializing XFRM netlink socket [ 3.565869] NET: Registered protocol family 10 [ 3.569719] Segment Routing with IPv6 [ 3.571421] NET: Registered protocol family 17 [ 3.573809] mpls_gso: MPLS GSO support [ 3.581786] RAS: Correctable Errors collector initialized. [ 3.583972] AVX version of gcm_enc/dec engaged. [ 3.585810] AES CTR mode by8 optimization enabled [ 3.669368] sched_clock: Marking stable (3669343195, 0)->(4540361154, -871017959) [ 3.672683] registered taskstats version 1 [ 3.674815] Loading compiled-in X.509 certificates [ 3.677312] zswap: loaded using pool lzo/zbud [ 3.705997] Key type big_key registered [ 3.720494] Key type encrypted registered [ 3.722543] ima: No TPM chip found, activating TPM-bypass! [ 3.724940] ima: Allocated hash algorithm: sha1 [ 3.726644] ima: No architecture policies found [ 3.728767] evm: Initialising EVM extended attributes: [ 3.730913] evm: security.selinux [ 3.732260] evm: security.ima [ 3.733520] evm: security.capability [ 3.735151] evm: HMAC attrs: 0x1 [ 3.739792] rtc_cmos 00:05: setting system clock to 2026-08-27 10:49:57 UTC (1787827797) [ 3.747321] debug: unmapping init [mem 0xffffffff91203000-0xffffffff913fffff] [ 3.750089] debug: unmapping init [mem 0xffffffff8ff82000-0xffffffff90258fff] [ 3.760087] Write protecting the kernel read-only data: 28672k [ 3.763509] debug: unmapping init [mem 0xffffffff8e603000-0xffffffff8e7fffff] [ 3.765940] debug: unmapping init [mem 0xffffffff8ef14000-0xffffffff8effffff] [ 3.803985] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.825234] systemd[1]: Detected virtualization kvm. [ 3.827880] systemd[1]: Detected architecture x86-64. [ 3.830414] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.856716] systemd[1]: No hostname configured. [ 3.858451] systemd[1]: Set hostname to . [ 3.861809] random: systemd: uninitialized urandom read (16 bytes read) [ 3.865262] systemd[1]: Initializing machine ID from random generator. [ 4.036821] random: systemd: uninitialized urandom read (16 bytes read) [ 4.041147] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.046501] random: systemd: uninitialized urandom read (16 bytes read) [ 4.050502] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.057925] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.805921] device-mapper: uevent: version 1.0.3 [ 4.809137] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices...[ 5.213690] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.544724] virtio_net virtio0 ens2: renamed from eth0 [ 5.612604] scsi host0: ata_piix [ 5.649467] scsi host1: ata_piix [ 5.651627] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.654134] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.327145] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.243420] random: crng init done [ 10.245491] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.749352] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.866087] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.124652] SELinux: Disabled at runtime. [ 12.193174] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.204177] systemd[1]: Detected virtualization kvm. [ 12.206263] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.699113] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.702363] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.710056] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.714476] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.718566] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.726511] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.731527] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 12.802686] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.107691] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.434116] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.442872] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.583854] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.605929] EDAC sbridge: Ver: 1.1.2 [ 15.408613] Key type dns_resolver registered [ 15.743475] NFS: Registering the id_resolver key type [ 15.747353] Key type id_resolver registered [ 15.749327] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg644-server login: [ 57.694667] hrtimer: interrupt took 9036333 ns [ 66.227700] libcfs: loading out-of-tree module taints kernel. [ 66.363267] Key type ._llcrypt registered [ 66.367705] Key type .llcrypt registered [ 66.525725] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 87.318358] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 89.177816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 89.200473] alg: No test for adler32 (adler32-zlib) [ 90.689970] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 91.699102] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 93.513482] Key type lgssc registered [ 95.442850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 113.760529] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 151.334534] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 163.320375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 163.340852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.645442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 164.689146] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 164.759213] Lustre: lustre-MDT0000: new disk, initializing [ 164.840818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 164.873183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 169.711506] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.070066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.187681] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 185.214895] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 185.221387] Lustre: Skipped 1 previous similar message [ 185.290286] Lustre: lustre-MDT0001: new disk, initializing [ 185.354624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 185.378797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 185.394144] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 189.896357] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.461968] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 203.771729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.070220] Lustre: lustre-OST0000: new disk, initializing [ 204.077130] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 204.086390] Lustre: 8450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.169423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.729781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.753084] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 207.841535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 210.653459] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.559920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.699640] Lustre: lustre-OST0001: new disk, initializing [ 225.703921] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 225.711844] Lustre: 9523:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.806759] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 232.693740] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.558671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 235.574242] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 235.660083] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 243.853413] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 254.703757] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 261.815693] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing check_logdir /tmp/testlogs/ [ 266.815158] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing yml_node [ 271.806948] Lustre: DEBUG MARKER: Client: 2.17.57.65 [ 274.300826] Lustre: DEBUG MARKER: MDS: 2.17.57.65 [ 276.499315] Lustre: DEBUG MARKER: OSS: 2.17.57.65 [ 277.949940] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 06:54:30 EDT 2026 [ 292.799619] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 294.128966] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 312.287968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.293147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 312.310418] Lustre: Skipped 1 previous similar message [ 312.325347] Lustre: Skipped 3 previous similar messages [ 315.664284] Lustre: server umount lustre-MDT0000 complete [ 318.956968] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828112 with bad export cookie 9991463039604220520 [ 318.962775] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 318.975356] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 322.528112] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 322.530649] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.531623] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 322.547983] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 322.573327] Lustre: Skipped 3 previous similar messages [ 323.553977] LustreError: 7743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.554619] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 323.573468] LustreError: 7743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 323.591243] Lustre: Skipped 2 previous similar messages [ 328.675079] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 328.676635] LustreError: 7743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.685901] Lustre: Skipped 1 previous similar message [ 328.727526] LustreError: 7743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 333.281785] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 333.783823] Lustre: server umount lustre-MDT0001 complete [ 337.746961] Lustre: server umount lustre-OST0000 complete [ 341.467840] Lustre: server umount lustre-OST0001 complete [ 349.412074] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 361.654586] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 393.614071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 393.833641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 393.856502] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 393.927422] Lustre: lustre-MDT0000: new disk, initializing [ 394.003428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.020803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 397.493453] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 407.165845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 407.259994] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 407.303290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 407.306633] Lustre: Skipped 1 previous similar message [ 407.387249] Lustre: lustre-MDT0001: new disk, initializing [ 407.478129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 407.513492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 407.520965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 411.796681] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 420.367382] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 425.777648] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 432.075684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 432.405163] Lustre: lustre-OST0000: new disk, initializing [ 432.408794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 432.414091] Lustre: 17047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 432.492355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 433.699099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 433.709556] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 433.809254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 438.485198] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 448.041142] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 452.452776] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 452.659418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 456.893552] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 457.187853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 459.235982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.257210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 459.263293] Lustre: Skipped 1 previous similar message [ 465.074551] Lustre: server umount lustre-OST0000 complete [ 468.960404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 468.970056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 468.984210] Lustre: Skipped 1 previous similar message [ 468.991675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 469.003144] Lustre: Skipped 2 previous similar messages [ 474.780688] Lustre: server umount lustre-MDT0000 complete [ 477.963675] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828271 with bad export cookie 9991463039604222053 [ 477.967118] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 477.972415] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 478.156453] Lustre: server umount lustre-MDT0001 complete [ 481.480580] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 06:57:53 (1787828273) [ 483.565783] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 485.077757] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 06:57:57 (1787828277) [ 491.030938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.361162] LustreError: 19455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.411978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.941065] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 496.616614] LustreError: 19456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.832115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 506.979788] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 514.762340] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 519.587943] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 524.762543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 530.067953] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 536.646699] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 546.784993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 546.793477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 546.811714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 546.823747] Lustre: Skipped 2 previous similar messages [ 546.845932] Lustre: Skipped 5 previous similar messages [ 550.141713] Lustre: server umount lustre-OST0000 complete [ 554.472016] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 557.028975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.051452] Lustre: Skipped 1 previous similar message [ 559.740570] Lustre: server umount lustre-MDT0000 complete [ 562.145091] LustreError: 19456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.174247] LustreError: 19456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 563.132300] LustreError: 19437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828356 with bad export cookie 9991463039604222928 [ 563.138727] LustreError: 19437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 563.143397] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 563.589741] Lustre: server umount lustre-MDT0001 complete [ 573.374606] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 575.866556] Key type lgssc unregistered [ 576.084683] LNet: 22988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 576.101756] LNetError: 22988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 576.122808] LNet: Removed LNI 192.168.206.144@tcp [ 577.069202] Key type .llcrypt unregistered [ 577.080927] Key type ._llcrypt unregistered [ 597.255479] Key type ._llcrypt registered [ 597.257564] Key type .llcrypt registered [ 597.348875] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 598.170853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 598.292924] alg: No test for adler32 (adler32-zlib) [ 599.311855] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 599.498360] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 601.135160] Key type lgssc registered [ 602.037708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 609.567575] loop: module loaded [ 609.676446] loop0: detected capacity change from 0 to 204800000 [ 609.969469] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 609.992265] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 610.305318] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 615.300399] loop0: detected capacity change from 0 to 204800000 [ 615.425615] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 615.439610] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 615.446519] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 619.411852] loop0: detected capacity change from 0 to 204800000 [ 623.966951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 623.988152] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.194247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 625.262586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 625.345854] Lustre: lustre-MDT0000: new disk, initializing [ 625.404105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 625.427954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 629.439549] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 637.299631] loop1: detected capacity change from 0 to 204800000 [ 642.179813] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 642.407672] Lustre: lustre-OST07c6: new disk, initializing [ 642.411564] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 642.414479] Lustre: Skipped 1 previous similar message [ 642.418310] Lustre: 26173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 642.479877] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 648.278093] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 652.394559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 652.418291] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 652.462427] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 658.435538] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 658.442759] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 658.448263] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 658.454586] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 658.460131] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 658.464872] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 677.753894] Lustre: 26690:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 677.765722] Lustre: 26690:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 677.777871] Lustre: 26690:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 677.788487] Lustre: 26690:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 677.795302] Lustre: 26690:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 677.800895] Lustre: 26690:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 683.372274] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 683.378969] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 683.383488] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 683.388194] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.419658] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 683.427278] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.434071] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 683.440216] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.445498] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 683.450497] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.455552] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 683.461171] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.710414] Lustre: 26690:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 686.728470] Lustre: 26690:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 686.741628] Lustre: 26690:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 686.749324] Lustre: 26690:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.762116] Lustre: 26690:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 686.772694] Lustre: 26690:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.781530] Lustre: 26690:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 686.790347] Lustre: 26690:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.796464] Lustre: 26690:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 686.817525] Lustre: 26690:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.838331] Lustre: 26690:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 686.858659] Lustre: 26690:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.892566] Lustre: 25160:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 691.910685] Lustre: 25160:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 691.926557] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 691.940768] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.954272] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 691.970955] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.985704] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 692.000182] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 692.006052] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 692.019229] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 692.028933] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 692.054554] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.934902] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 699.939041] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 699.944376] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 699.952428] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 699.954916] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 699.958898] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 703.193369] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 703.207932] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 703.216979] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 703.226250] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 703.235797] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 703.243658] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 754.657870] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 754.669441] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 757.426223] Lustre: server umount lustre-OST07c6 complete [ 765.905602] Lustre: server umount lustre-MDT0000 complete [ 781.370771] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 783.812762] Key type lgssc unregistered [ 784.037762] LNet: 28751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.046275] LNetError: 28751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.072177] LNet: Removed LNI 192.168.206.144@tcp [ 784.963643] Key type .llcrypt unregistered [ 784.966102] Key type ._llcrypt unregistered [ 801.746596] Key type ._llcrypt registered [ 801.749351] Key type .llcrypt registered [ 801.837963] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 802.723345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 802.736155] alg: No test for adler32 (adler32-zlib) [ 803.735851] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 804.037290] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 805.719168] Key type lgssc registered [ 806.909294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 814.368651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 814.396852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 815.961995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 820.604507] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 825.318973] LustreError: 30020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 825.336304] LustreError: 30020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 829.212987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 829.676409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 833.822441] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 840.860613] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 845.776827] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 851.647861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 852.013056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 857.490348] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 861.244387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 864.223660] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 880.632314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 880.642822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.660685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 881.632375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 881.638180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 881.646452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.563799] Lustre: server umount lustre-OST0000 complete [ 891.362726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 891.371969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 891.396061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 891.403783] Lustre: Skipped 1 previous similar message [ 894.232441] Lustre: server umount lustre-MDT0000 complete [ 896.996800] LustreError: 30021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 897.016970] LustreError: 30021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 898.007135] LustreError: 30003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828691 with bad export cookie 10413920535209757485 [ 898.014943] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 898.033574] LustreError: 30003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 898.318948] Lustre: server umount lustre-MDT0001 complete [ 909.698968] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 913.269933] Key type lgssc unregistered [ 913.731467] LNet: 33758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.741781] LNetError: 33758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.755961] LNet: Removed LNI 192.168.206.144@tcp [ 914.842692] Key type .llcrypt unregistered [ 914.844976] Key type ._llcrypt unregistered [ 935.360152] Key type ._llcrypt registered [ 935.362079] Key type .llcrypt registered [ 935.469877] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 936.566820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 936.635421] alg: No test for adler32 (adler32-zlib) [ 937.811785] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 938.128195] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 939.815194] Key type lgssc registered [ 940.946329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 948.901391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 948.919355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 950.338314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 955.042212] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 960.496877] LustreError: 35029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 960.546777] LustreError: 35029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 965.604504] LustreError: 35030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 965.693532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 966.011586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 970.331298] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 977.953377] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 983.835878] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 990.771598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 991.182848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 995.358223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 997.544674] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1004.675940] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1020.900161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1020.901830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1020.904494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1020.906561] LustreError: Skipped 1 previous similar message [ 1020.915238] Lustre: Skipped 1 previous similar message [ 1023.992317] Lustre: server umount lustre-OST0000 complete [ 1027.559746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1027.569096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1027.583449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1027.587445] Lustre: Skipped 1 previous similar message [ 1031.140164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1031.142261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1031.165705] Lustre: Skipped 1 previous similar message [ 1032.970802] Lustre: server umount lustre-MDT0000 complete [ 1036.104977] LustreError: 35011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828829 with bad export cookie 18046746763637474192 [ 1036.115133] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1036.119304] LustreError: 35011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1036.576625] Lustre: server umount lustre-MDT0001 complete [ 1048.597107] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 1052.210706] Key type lgssc unregistered [ 1052.670471] LNet: 38588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1052.680616] LNetError: 38588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1052.727784] LNet: Removed LNI 192.168.206.144@tcp [ 1053.966749] Key type .llcrypt unregistered [ 1053.968827] Key type ._llcrypt unregistered [ 1074.849253] Key type ._llcrypt registered [ 1074.851699] Key type .llcrypt registered [ 1074.945401] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 1075.704511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1075.718304] alg: No test for adler32 (adler32-zlib) [ 1076.735453] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 1076.903179] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 1078.607174] Key type lgssc registered [ 1079.913350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1087.853557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1087.869209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.432458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1093.964823] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1099.748645] LustreError: 40000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1099.771530] LustreError: 40000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1102.710332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.003559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1106.778849] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1113.864977] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1118.947209] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1125.409286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1125.734611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1131.419963] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1134.961772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1138.529755] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1145.316347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1145.322973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1145.326878] Lustre: Skipped 1 previous similar message [ 1145.335633] Lustre: Skipped 1 previous similar message [ 1146.181904] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.44@tcp (stopping) [ 1147.140937] Lustre: server umount lustre-MDT0000 complete [ 1150.439142] LustreError: 42577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1150.451699] LustreError: 42577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1150.859053] LustreError: 40734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828944 with bad export cookie 8605395451595678512 [ 1150.862462] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1150.870792] LustreError: 40734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1151.264517] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.44@tcp (stopping) [ 1155.553215] LustreError: 39996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1155.554525] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1155.569028] LustreError: 39996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1155.579055] Lustre: Skipped 1 previous similar message [ 1155.580029] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1160.672427] LustreError: 39995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.683706] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1160.706755] LustreError: 39995:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1160.712950] Lustre: Skipped 1 previous similar message [ 1165.279339] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1165.546344] Lustre: server umount lustre-MDT0001 complete [ 1180.344819] Lustre: server umount lustre-OST0000 complete [ 1195.719598] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 1198.495449] Key type lgssc unregistered [ 1198.752701] LNet: 43860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1198.760642] LNetError: 43860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1198.778151] LNet: Removed LNI 192.168.206.144@tcp [ 1199.578169] Key type .llcrypt unregistered [ 1199.585463] Key type ._llcrypt unregistered [ 1222.480186] Key type ._llcrypt registered [ 1222.482720] Key type .llcrypt registered [ 1222.576536] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 1223.784701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1223.824100] alg: No test for adler32 (adler32-zlib) [ 1225.005709] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 1225.385854] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 1227.079582] Key type lgssc registered [ 1228.437160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1236.153813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1236.168940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.753841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.326539] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1248.226552] LustreError: 45278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.243303] LustreError: 45278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1251.397594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.784871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1256.799445] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1265.821651] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1270.827863] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1277.647154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1278.052797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1283.199864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1283.904843] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1291.178723] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1301.990744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1301.992132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1302.010390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1302.028843] Lustre: Skipped 1 previous similar message [ 1302.040807] Lustre: Skipped 1 previous similar message [ 1303.450763] Lustre: server umount lustre-OST0000 complete [ 1323.490201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1323.495995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.544536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1326.566727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1326.593770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1331.680655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1331.688447] Lustre: Skipped 2 previous similar messages [ 1336.289213] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1336.803908] LustreError: 45278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1336.835458] LustreError: 45278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1336.889991] Lustre: server umount lustre-MDT0000 complete [ 1340.787536] LustreError: 45257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829134 with bad export cookie 4001565193627454038 [ 1340.803506] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1340.807235] LustreError: 45257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1355.231137] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1355.714676] Lustre: server umount lustre-MDT0001 complete [ 1368.060680] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 1371.140320] Key type lgssc unregistered [ 1371.497767] LNet: 48993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1371.511799] LNetError: 48993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.533769] LNet: Removed LNI 192.168.206.144@tcp [ 1372.503123] Key type .llcrypt unregistered [ 1372.506282] Key type ._llcrypt unregistered [ 1398.129684] Key type ._llcrypt registered [ 1398.133549] Key type .llcrypt registered [ 1398.247740] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 1400.042432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1400.109474] alg: No test for adler32 (adler32-zlib) [ 1401.065069] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 1401.291677] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 1402.919174] Key type lgssc registered [ 1404.490082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1413.357466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1413.386452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.119838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1420.072124] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1424.365350] LustreError: 50411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1424.390240] LustreError: 50411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1428.914674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1429.253340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1435.216522] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1442.135334] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1447.500398] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1453.755528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1453.995534] LustreError: 52232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1454.087293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1459.890237] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1460.218759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1467.351880] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1473.332692] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1476.577395] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1488.723945] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-27 7h14m39s [ 1490.205767] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1495.842673] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.44@tcp (stopping) [ 1496.039512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1496.053896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1496.071151] Lustre: Skipped 1 previous similar message [ 1496.901531] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.44@tcp (stopping) [ 1496.907941] Lustre: Skipped 3 previous similar messages [ 1498.390717] Lustre: server umount lustre-MDT0000 complete [ 1500.929681] LustreError: 50407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.538438] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1503.205256] LustreError: 50406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1503.224810] LustreError: 50406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1507.439623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1507.604497] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1507.890111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1507.979363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1508.917573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1512.665409] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1516.244476] Lustre: DEBUG MARKER: Wait for df (36764) ... [ 1518.208759] Lustre: DEBUG MARKER: done [ 1523.168089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1523.179642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.193644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1523.200560] Lustre: Skipped 3 previous similar messages [ 1524.196363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1528.383778] Lustre: server umount lustre-OST0000 complete [ 1534.439950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1534.447155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1534.460312] Lustre: Skipped 1 previous similar message [ 1534.476095] Lustre: Skipped 3 previous similar messages [ 1539.223432] Lustre: server umount lustre-MDT0000 complete [ 1541.602412] LustreError: 50412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1541.619569] LustreError: 50412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1542.789810] LustreError: 53038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829336 with bad export cookie 2831958757682033056 [ 1542.801353] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1542.801533] LustreError: 53038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1543.327648] Lustre: server umount lustre-MDT0001 complete [ 1554.286441] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 1557.120652] Key type lgssc unregistered [ 1557.396673] LNet: 55338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.405480] LNetError: 55338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.417319] LNet: Removed LNI 192.168.206.144@tcp [ 1558.256095] Key type .llcrypt unregistered [ 1558.258310] Key type ._llcrypt unregistered [ 1594.751235] Key type ._llcrypt registered [ 1594.755585] Key type .llcrypt registered [ 1594.912075] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 1596.025513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1596.083450] alg: No test for adler32 (adler32-zlib) [ 1597.127809] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 1597.389930] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 1599.135230] Key type lgssc registered [ 1600.289412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1607.660671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1607.673945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1608.940586] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1609.016317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1609.232537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1613.680128] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1621.973893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1622.127093] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1622.183703] Lustre: 57526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1622.524241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1627.171598] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1634.195922] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1640.287426] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1647.309160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1647.565808] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1647.794751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1654.588435] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1657.503263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1662.222215] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1668.255319] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-27 7h17m40s [ 1669.950144] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1677.023343] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1680.067666] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1680.123958] LustreError: 57534:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1700.645491] Lustre: lustre-MDT0000: Client e60f0716-dbb1-42c3-9120-915de45a327b (at 192.168.206.44@tcp) reconnecting [ 1720.143241] LustreError: 57534:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1722.594674] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1724.949544] Lustre: DEBUG MARKER: done [ 1734.112355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1734.123587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1734.145517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1734.638983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1734.659808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1734.678868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1737.478548] Lustre: server umount lustre-OST0000 complete [ 1744.864887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1744.878776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1744.884018] Lustre: Skipped 1 previous similar message [ 1747.737235] Lustre: server umount lustre-MDT0000 complete [ 1751.782499] LustreError: 57518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829545 with bad export cookie 14425921668182289518 [ 1751.787239] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1751.794759] LustreError: 57518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1752.279625] Lustre: server umount lustre-MDT0001 complete [ 1763.543337] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 1766.424264] Key type lgssc unregistered [ 1766.692792] LNet: 61857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1766.705396] LNetError: 61857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1767.787537] LNet: Removed LNI 192.168.206.144@tcp [ 1768.796245] Key type .llcrypt unregistered [ 1768.798611] Key type ._llcrypt unregistered [ 1816.640094] Key type ._llcrypt registered [ 1816.645642] Key type .llcrypt registered [ 1816.817364] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 1818.120430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1818.542770] alg: No test for adler32 (adler32-zlib) [ 1819.847467] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 1820.208273] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 1821.975202] Key type lgssc registered [ 1823.380955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1827.332878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1827.371191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.821368] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1828.862584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1829.084264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.478171] Lustre: Failing over lustre-MDT0000 [ 1830.778295] Lustre: server umount lustre-MDT0000 complete [ 1838.701634] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 07:20:29 (1787829629) [ 1846.497090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1846.997760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1852.410784] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1861.400659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.518906] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1861.567642] Lustre: 64663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1862.004094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1867.352226] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1875.649218] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1881.530443] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1889.095470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1889.378705] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1889.630716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1895.999758] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1899.127679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1903.292217] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1909.453793] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1918.432970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1918.442931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.456529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1919.972127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.973562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1920.034366] Lustre: Skipped 1 previous similar message [ 1922.533542] Lustre: server umount lustre-MDT0000 complete [ 1927.229829] LustreError: 64653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829720 with bad export cookie 7478654547141100825 [ 1927.245224] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1927.248234] LustreError: 64653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1927.782660] Lustre: server umount lustre-MDT0001 complete [ 1932.101498] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1937.208775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1945.584453] Lustre: 63584:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829723/real 1787829723] req@000000009ff544d5 x1874675224321152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787829739 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1945.630830] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.666352] Lustre: Skipped 1 previous similar message [ 1945.884682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.768556] Lustre: 63584:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829726/real 1787829726] req@000000000e6c27de x1874675224321280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787829742 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1951.711157] Lustre: 63586:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829729/real 1787829729] req@00000000f6f1e70e x1874675224321408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787829745 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1952.867162] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x67c984afe0348dfc [ 1952.893135] Lustre: MGC192.168.206.144@tcp: Connection restored to 0@lo (at 0@lo) [ 1953.285921] LustreError: 66510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.311587] LustreError: 66510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1953.435566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.559152] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1954.401062] Lustre: 63586:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829731/real 1787829731] req@000000001651feb4 x1874675224321664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787829747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1954.468570] LustreError: 66511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1955.490580] LustreError: 68433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1955.526093] LustreError: 68433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1958.496709] Lustre: 63584:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829736/real 1787829736] req@00000000f397225b x1874675224321792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787829752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1958.661150] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1960.937964] LustreError: 66510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1960.964935] LustreError: 66510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1966.055614] LustreError: 68564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1966.080691] LustreError: 68564:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1969.371888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.906197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1975.024391] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1975.556932] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 1984.497817] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1991.375446] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2000.874528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.877587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2000.882056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2000.882064] Lustre: Skipped 1 previous similar message [ 2000.889750] Lustre: Skipped 1 previous similar message [ 2003.908283] Lustre: server umount lustre-MDT0000 complete [ 2005.986537] LustreError: 68435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2006.019462] LustreError: 68435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2007.399266] LustreError: 64655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829801 with bad export cookie 7478654547141103100 [ 2007.412333] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2007.418969] LustreError: 64655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2007.758660] Lustre: server umount lustre-MDT0001 complete [ 2014.445713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2024.669495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2026.473798] Lustre: 63583:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829804/real 1787829804] req@000000009ef02a53 x1874675224349824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787829820 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2026.517617] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.532750] Lustre: Skipped 1 previous similar message [ 2032.674715] LustreError: 63582:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000082c8cbed x1874675224350848/t0(0) o250->MGC192.168.206.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2032.959075] LustreError: 66511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2033.037367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2033.072388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2038.429850] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2049.367059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.579583] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2055.145521] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2055.152057] Lustre: Skipped 1 previous similar message [ 2062.345469] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2068.648969] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2076.644090] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2076.647656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2076.661778] Lustre: Skipped 2 previous similar messages [ 2076.675988] Lustre: Skipped 4 previous similar messages [ 2081.766820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2081.776820] Lustre: Skipped 2 previous similar messages [ 2082.284922] Lustre: server umount lustre-MDT0000 complete [ 2086.357079] LustreError: 67575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829880 with bad export cookie 7478654547141104479 [ 2086.362382] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2086.367889] LustreError: 67575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2086.685843] Lustre: server umount lustre-MDT0001 complete [ 2093.081478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2096.376604] Lustre: DEBUG MARKER: files compared the same [ 2109.034237] Lustre: server umount lustre-OST0000 complete [ 2123.527270] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 2125.839226] Key type lgssc unregistered [ 2126.094550] LNet: 74395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2126.103896] LNetError: 74395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2126.121853] LNet: Removed LNI 192.168.206.144@tcp [ 2126.955856] Key type .llcrypt unregistered [ 2126.959844] Key type ._llcrypt unregistered [ 2147.534793] Key type ._llcrypt registered [ 2147.536742] Key type .llcrypt registered [ 2147.620413] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 2148.338674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2148.405606] alg: No test for adler32 (adler32-zlib) [ 2149.450873] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 2149.674935] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 2151.335238] Key type lgssc registered [ 2152.537371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2159.826529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2159.842405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.273163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.721836] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug +malloc all [ 2170.855254] LustreError: 75791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.870982] LustreError: 75791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2174.389801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2174.704590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2179.157285] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug +malloc all [ 2186.137332] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2191.069848] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2198.965280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2199.251038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.704711] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug +malloc all [ 2205.422367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2210.983486] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2221.025326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2221.036724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2221.047452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2226.145144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2226.159959] Lustre: Skipped 1 previous similar message [ 2231.266716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2231.273269] Lustre: Skipped 1 previous similar message [ 2233.311402] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2233.459250] Lustre: server umount lustre-OST0000 complete [ 2241.506577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2241.507040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2241.538774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2241.544772] Lustre: Skipped 2 previous similar messages [ 2241.566029] Lustre: Skipped 2 previous similar messages [ 2243.632559] Lustre: server umount lustre-MDT0000 complete [ 2246.632126] LustreError: 76810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2246.651453] LustreError: 76810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2247.182916] LustreError: 75772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830040 with bad export cookie 1918539874955028827 [ 2247.188727] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2247.196759] LustreError: 75772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2247.503991] Lustre: server umount lustre-MDT0001 complete [ 2257.846234] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 2260.149737] Key type lgssc unregistered [ 2260.420743] LNet: 79327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2260.432819] LNetError: 79327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2260.451844] LNet: Removed LNI 192.168.206.144@tcp [ 2261.054322] Key type .llcrypt unregistered [ 2261.056858] Key type ._llcrypt unregistered [ 2282.848807] Key type ._llcrypt registered [ 2282.851085] Key type .llcrypt registered [ 2282.961250] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 2284.107548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2284.134971] alg: No test for adler32 (adler32-zlib) [ 2285.254714] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 2285.550748] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 2287.215155] Key type lgssc registered [ 2288.392418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2295.752817] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2295.771817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2311.263443] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2316.385875] LustreError: 80716:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.144@tcp: failed processing log, type 4: rc = -110 [ 2343.071699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.796844] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2358.658857] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2368.313537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2368.539343] Lustre: *** cfs_fail_loc=706, val=0*** [ 2368.988186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.098252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2374.118071] LustreError: 80741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.143087] LustreError: 80741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2375.115382] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2378.246494] LustreError: 81803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.286583] LustreError: 81803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2383.335381] LustreError: 80742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.378153] LustreError: 80742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2384.707902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2385.103725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2390.239702] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2399.607785] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2406.477747] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2409.954081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2409.979262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2410.005556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2410.991394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2411.011986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2411.017993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2415.436082] Lustre: server umount lustre-OST0000 complete [ 2421.216433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2421.235808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.246391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2421.279364] Lustre: Skipped 1 previous similar message [ 2421.279584] Lustre: Skipped 1 previous similar message [ 2426.246610] Lustre: server umount lustre-MDT0000 complete [ 2426.336241] LustreError: 81803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.364680] LustreError: 81803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2430.434765] LustreError: 81808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.445934] LustreError: 81808:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2430.544872] LustreError: 80722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830224 with bad export cookie 10719101574175561653 [ 2430.553334] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2430.896774] Lustre: server umount lustre-MDT0001 complete [ 2442.149532] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 2445.222825] Key type lgssc unregistered [ 2445.641706] LNet: 84337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2445.649290] LNetError: 84337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2445.680832] LNet: Removed LNI 192.168.206.144@tcp [ 2446.793442] Key type .llcrypt unregistered [ 2446.795929] Key type ._llcrypt unregistered [ 2470.796229] Key type ._llcrypt registered [ 2470.797902] Key type .llcrypt registered [ 2470.910374] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 2471.811594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2471.830844] alg: No test for adler32 (adler32-zlib) [ 2472.894929] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 2473.066742] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 2474.703153] Key type lgssc registered [ 2475.662927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2482.889887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2482.923361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.978500] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2496.284906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.806238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2502.113880] LustreError: 86442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.128616] LustreError: 86442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2502.320897] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2507.232765] LustreError: 86445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.258799] LustreError: 86445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2511.120949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2511.416421] LustreError: 86442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.450271] LustreError: 86442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2511.465772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2516.450710] LustreError: 86446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2516.483534] LustreError: 86446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2517.919932] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2521.572493] LustreError: 86441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2521.606991] LustreError: 86441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2526.775163] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2526.796847] Lustre: MGS: Client 9e56760e-d8e7-443d-bbe7-397d9f7fb06a (at 0@lo) reconnecting [ 2526.833423] LustreError: 86456:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2527.199330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2527.294861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2531.964790] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2547.703891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2547.715132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2547.729872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2547.753096] Lustre: Skipped 1 previous similar message [ 2550.593752] Lustre: server umount lustre-OST0000 complete [ 2557.922838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2557.923415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2557.924546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2557.948571] Lustre: Skipped 2 previous similar messages [ 2560.624136] Lustre: server umount lustre-MDT0000 complete [ 2563.039748] LustreError: 87986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.055281] LustreError: 87986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2565.033949] Lustre: server umount lustre-MDT0001 complete [ 2569.637902] Lustre: server umount MGS complete [ 2580.907464] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 07:32:52 (1787830372) [ 2613.018563] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 2624.692351] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 2662.966582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.679498] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2680.830216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.162895] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2681.245716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2681.378619] Lustre: lustre-MDT0001: new disk, initializing [ 2681.535803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2681.565620] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2681.581544] Lustre: 93613:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2681.603957] LustreError: 93613:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2685.876822] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2695.955716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2696.156753] Lustre: lustre-OST0000: new disk, initializing [ 2696.160217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2696.167479] Lustre: 94379:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2696.213252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2701.613230] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2702.944323] Lustre: 94747:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000003d1ff767 x1874675909042048/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787830512 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2710.318756] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2717.111703] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2717.127436] Lustre: MGS: Client 229a8844-9de1-46f2-a5c7-54508fe8f342 (at 0@lo) reconnecting [ 2717.142554] LustreError: 93599:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2717.179966] Lustre: 95424:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2717.213478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2717.283798] Lustre: lustre-MDT0000: new disk, initializing [ 2717.463727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2717.516625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2717.543420] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2718.627395] Lustre: 95449:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000009b309047 x1874675909048576/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787830528 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2722.352850] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2723.744125] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2727.721952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2727.731993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2727.811505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2753.509656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2753.522096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.543257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2753.548362] Lustre: Skipped 1 previous similar message [ 2755.555312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.558666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2755.587364] Lustre: Skipped 1 previous similar message [ 2758.776090] Lustre: server umount lustre-OST0000 complete [ 2765.795133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2765.816954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.818329] Lustre: Skipped 1 previous similar message [ 2769.038340] Lustre: server umount lustre-MDT0000 complete [ 2770.919167] LustreError: 93589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.939364] LustreError: 93589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2787.295894] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2787.580973] Lustre: server umount lustre-MDT0001 complete [ 2790.857629] Lustre: server umount MGS complete [ 2801.846798] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 07:36:33 (1787830593) [ 2817.977619] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 2820.834499] Key type lgssc unregistered [ 2821.094274] LNet: 97952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2821.104727] LNetError: 97952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2821.125035] LNet: Removed LNI 192.168.206.144@tcp [ 2821.928688] Key type .llcrypt unregistered [ 2821.933630] Key type ._llcrypt unregistered [ 2837.327328] Key type ._llcrypt registered [ 2837.331299] Key type .llcrypt registered [ 2837.432440] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 2838.520261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2838.596042] alg: No test for adler32 (adler32-zlib) [ 2839.622436] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 2839.818260] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 2841.495163] Key type lgssc registered [ 2842.703416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2850.188479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2850.217857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2851.706631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2856.241391] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2862.051786] LustreError: 99168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2862.075390] LustreError: 99168:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2864.923300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2865.402386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2869.561363] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2877.288475] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2883.596883] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2890.221612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2890.713623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.730690] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 2896.881795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2903.899412] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2912.224468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2912.228708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2912.239183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2912.279210] Lustre: Skipped 1 previous similar message [ 2917.349090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2917.356568] Lustre: Skipped 1 previous similar message [ 2918.141466] Lustre: server umount lustre-OST0000 complete [ 2921.952753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2921.962476] LustreError: Skipped 1 previous similar message [ 2921.976243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.997526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2922.005637] Lustre: Skipped 1 previous similar message [ 2927.588051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2927.599449] Lustre: Skipped 3 previous similar messages [ 2928.171222] Lustre: server umount lustre-MDT0000 complete [ 2932.705541] LustreError: 99162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2932.718046] LustreError: 99162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2933.338682] LustreError: 99148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830727 with bad export cookie 14020027445184675098 [ 2933.340214] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2933.348677] LustreError: 99148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2933.604455] Lustre: server umount lustre-MDT0001 complete [ 2948.185530] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 2952.154878] Key type lgssc unregistered [ 2952.601743] LNet: 102702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2952.623723] LNetError: 102702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2952.662678] LNet: Removed LNI 192.168.206.144@tcp [ 2953.735397] Key type .llcrypt unregistered [ 2953.741358] Key type ._llcrypt unregistered [ 2970.888285] Key type ._llcrypt registered [ 2970.891756] Key type .llcrypt registered [ 2971.038556] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 2972.457359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2972.553370] alg: No test for adler32 (adler32-zlib) [ 2973.640761] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 2973.936714] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 2975.679766] Key type lgssc registered [ 2976.768811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2983.098076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2983.107809] LustreError: 103770:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2983.247161] LustreError: 103790:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2983.252550] LustreError: 103770:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4862 [ 2983.261722] LustreError: 103770:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2983.265906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.271442] LustreError: 103770:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2984.670479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2991.819073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2992.050062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2996.052659] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3001.322578] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing lsmod [ 3004.336606] LustreError: 104985:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3004.383987] LustreError: 104995:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3004.387841] LustreError: 104985:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4957 [ 3004.392500] LustreError: 104985:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3004.399703] LustreError: 104985:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3004.403429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3004.834940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.467140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3007.485105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.509279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.022840] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.023593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.046113] Lustre: Skipped 1 previous similar message [ 3010.051625] Lustre: Skipped 2 previous similar messages [ 3013.818805] Lustre: server umount lustre-MDT0000 complete [ 3015.140309] LustreError: 105031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3015.169092] LustreError: 105031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3017.192410] LustreError: 103827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3017.229144] LustreError: 103827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3018.024630] LustreError: 103808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830811 with bad export cookie 17189745819804644765 [ 3018.052271] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3018.079924] Lustre: 103819:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3018.458328] Lustre: server umount lustre-MDT0001 complete [ 3033.442173] Lustre: server umount lustre-OST0000 complete [ 3041.893333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3042.323592] LustreError: 106057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.394106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.205748] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3047.403721] LustreError: 106058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.523429] LustreError: 106057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3055.541757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.934732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3060.250333] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3068.245692] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3073.505804] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3079.968231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3080.312699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3085.457556] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3087.480552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3093.101247] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3102.186939] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3102.196171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3102.214053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3103.205823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3107.567968] Lustre: server umount lustre-OST0000 complete [ 3112.417215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3112.424530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3112.444260] Lustre: Skipped 1 previous similar message [ 3112.451963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3112.463162] Lustre: Skipped 2 previous similar messages [ 3118.561850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3118.566398] Lustre: Skipped 3 previous similar messages [ 3126.241631] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3126.534669] Lustre: server umount lustre-MDT0000 complete [ 3128.800779] LustreError: 106058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3128.813791] LustreError: 106058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3130.473852] LustreError: 106038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830924 with bad export cookie 17189745819804645444 [ 3130.478354] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3130.490713] LustreError: 106038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3130.800574] Lustre: server umount lustre-MDT0001 complete [ 3141.353438] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 3143.860976] Key type lgssc unregistered [ 3144.176541] LNet: 109598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3144.184080] LNetError: 109598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3144.228348] LNet: Removed LNI 192.168.206.144@tcp [ 3145.002799] Key type .llcrypt unregistered [ 3145.006307] Key type ._llcrypt unregistered [ 3165.369817] Key type ._llcrypt registered [ 3165.371986] Key type .llcrypt registered [ 3165.477722] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 3166.533841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3166.580338] alg: No test for adler32 (adler32-zlib) [ 3167.637536] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 3167.891179] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 3169.567165] Key type lgssc registered [ 3170.833569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3177.974918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3178.004764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3179.551412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3183.854148] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3189.732709] LustreError: 110994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3189.749269] LustreError: 110994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3192.075340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3192.605822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3197.383810] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3205.273386] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3211.374111] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3218.331445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3218.750839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.892647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3226.553321] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3235.143133] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3243.891347] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3247.334724] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3253.738716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3253.744691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3253.769284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3253.787169] Lustre: Skipped 1 previous similar message [ 3253.806110] Lustre: Skipped 1 previous similar message [ 3257.477783] Lustre: server umount lustre-OST0000 complete [ 3263.981919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3264.000946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3264.011480] Lustre: Skipped 1 previous similar message [ 3267.811981] Lustre: server umount lustre-MDT0000 complete [ 3269.089458] LustreError: 112386:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3269.115558] LustreError: 112386:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3271.614205] LustreError: 110975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831065 with bad export cookie 5452292099941970582 [ 3271.624130] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3271.629563] LustreError: 110975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3271.957264] Lustre: server umount lustre-MDT0001 complete [ 3283.335841] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 3286.051102] Key type lgssc unregistered [ 3286.395857] LNet: 114635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3286.407696] LNetError: 114635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3286.440786] LNet: Removed LNI 192.168.206.144@tcp [ 3287.263737] Key type .llcrypt unregistered [ 3287.266735] Key type ._llcrypt unregistered [ 3303.471760] Key type ._llcrypt registered [ 3303.474266] Key type .llcrypt registered [ 3303.612900] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 3304.532550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3304.548135] alg: No test for adler32 (adler32-zlib) [ 3305.731208] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 3306.012243] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 3307.703168] Key type lgssc registered [ 3308.920257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3316.994229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3317.009887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3318.495453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3323.146367] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3328.996158] LustreError: 115877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3329.026601] LustreError: 115877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3331.887034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3332.372830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3336.826451] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3344.841092] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3350.466725] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3356.150302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3356.471254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3362.520452] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3363.638937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3368.804747] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3378.144213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3378.156802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3378.172068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3379.183215] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3379.183610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3379.183909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3382.927883] Lustre: server umount lustre-OST0000 complete [ 3388.902639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3388.909493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3388.924163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3388.933872] Lustre: Skipped 1 previous similar message [ 3392.867588] Lustre: server umount lustre-MDT0000 complete [ 3394.530238] LustreError: 117117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3394.553828] LustreError: 117117:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3396.104954] LustreError: 115858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831189 with bad export cookie 13895108026266090294 [ 3396.114177] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3396.117673] LustreError: 115858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3396.602737] Lustre: server umount lustre-MDT0001 complete [ 3406.385747] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 3409.004772] Key type lgssc unregistered [ 3409.281408] LNet: 119412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3409.290392] LNetError: 119412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3409.307626] LNet: Removed LNI 192.168.206.144@tcp [ 3410.163563] Key type .llcrypt unregistered [ 3410.168419] Key type ._llcrypt unregistered [ 3430.909480] Key type ._llcrypt registered [ 3430.916260] Key type .llcrypt registered [ 3431.017321] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 3431.922872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3431.945495] alg: No test for adler32 (adler32-zlib) [ 3433.107469] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 3433.359657] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 3434.999176] Key type lgssc registered [ 3435.883445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3442.245884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3442.271586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3443.665293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3447.276971] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3453.922570] LustreError: 120819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3453.934497] LustreError: 120819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3454.592340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3454.984252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3459.132445] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3466.616819] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3471.655412] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3478.587563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3478.911059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3484.766236] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3489.139911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3491.022501] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3496.885377] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3500.323739] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3500.328405] Lustre: Skipped 2 previous similar messages [ 3504.441665] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3504.448767] Lustre: Skipped 2 previous similar messages [ 3505.182585] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3505.205974] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3515.702859] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.44@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3515.710656] Lustre: Skipped 2 previous similar messages [ 3522.597507] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.44@tcp 192.168.0.[2,10]@tcp [ 3525.517931] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.44/24@tcp in log lustre-MDT0000 [ 3525.529041] Lustre: Skipped 2 previous similar messages [ 3532.400629] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.44/24@tcp [ 3532.423317] Lustre: Skipped 1 previous similar message [ 3538.922332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3538.923219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3538.929054] LustreError: Skipped 2 previous similar messages [ 3538.960187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3540.451181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3540.454013] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3544.080265] Lustre: server umount lustre-OST0000 complete [ 3550.690581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3550.716520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3550.722688] Lustre: Skipped 1 previous similar message [ 3554.436855] Lustre: server umount lustre-MDT0000 complete [ 3555.818104] LustreError: 120819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3555.859092] LustreError: 120819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3559.053136] LustreError: 120801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831352 with bad export cookie 2368405174646103122 [ 3559.067887] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3559.491923] Lustre: server umount lustre-MDT0001 complete [ 3569.573593] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 3572.670374] Key type lgssc unregistered [ 3572.927461] LNet: 125707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3572.935046] LNetError: 125707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3573.999105] LNet: Removed LNI 192.168.206.144@tcp [ 3574.941351] Key type .llcrypt unregistered [ 3574.943107] Key type ._llcrypt unregistered [ 3602.453104] Key type ._llcrypt registered [ 3602.458413] Key type .llcrypt registered [ 3602.549566] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 3603.537079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3603.671652] alg: No test for adler32 (adler32-zlib) [ 3604.769124] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 3605.089283] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 3606.759298] Key type lgssc registered [ 3607.707795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3614.434140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3614.463286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3615.872504] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3615.919658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3615.923639] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.44/24@tcp [ 3620.097133] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3625.453692] LustreError: 127268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3625.479483] LustreError: 127268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3627.621044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.923263] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3627.969824] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.44/24@tcp [ 3627.995228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3632.216956] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3638.356837] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3642.614509] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3648.659287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3648.954676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3653.548848] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3659.268482] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3661.227731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3673.076623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3673.087630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.108749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3674.593797] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3674.601793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3674.614450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3676.642886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3676.649025] Lustre: Skipped 1 previous similar message [ 3678.422322] Lustre: server umount lustre-OST0000 complete [ 3684.321093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3684.328227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3684.341995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3686.881689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3688.564266] Lustre: server umount lustre-MDT0000 complete [ 3692.001751] LustreError: 127248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831485 with bad export cookie 8244107226331995524 [ 3692.006399] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3692.020080] LustreError: 127248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3692.391990] Lustre: server umount lustre-MDT0001 complete [ 3702.628582] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 3705.161994] Key type lgssc unregistered [ 3705.486301] LNet: 130860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3705.501928] LNetError: 130860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3705.525082] LNet: Removed LNI 192.168.206.144@tcp [ 3706.388686] Key type .llcrypt unregistered [ 3706.391229] Key type ._llcrypt unregistered [ 3731.069154] Key type ._llcrypt registered [ 3731.072536] Key type .llcrypt registered [ 3731.181727] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 3743.747837] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 3744.550677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3744.721105] alg: No test for adler32 (adler32-zlib) [ 3745.789326] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 3746.102208] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 3747.775177] Key type lgssc registered [ 3748.626756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3780.220747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3780.254263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3781.462927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3781.483117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3781.555349] Lustre: lustre-MDT0000: new disk, initializing [ 3781.643095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3781.660101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3786.043962] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3795.685165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3795.811754] Lustre: 134241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3795.858694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3795.864827] Lustre: Skipped 1 previous similar message [ 3795.931340] Lustre: lustre-MDT0001: new disk, initializing [ 3796.032885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3796.059512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3796.066266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3800.100818] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3808.649754] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3814.610673] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3821.136900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3821.329282] Lustre: lustre-OST0000: new disk, initializing [ 3821.332232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3821.337100] Lustre: 136207:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3821.404978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3827.620720] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3830.341783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3830.354475] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3830.432065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3837.115717] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3842.082611] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3842.289233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3846.578503] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3846.818039] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3851.242430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3851.246428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3851.250639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3851.258464] Lustre: Skipped 1 previous similar message [ 3851.276302] Lustre: Skipped 1 previous similar message [ 3854.731602] Lustre: server umount lustre-OST0000 complete [ 3861.473291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3861.490824] Lustre: Skipped 1 previous similar message [ 3861.493559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3861.504500] Lustre: Skipped 1 previous similar message [ 3866.598494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3866.606633] Lustre: Skipped 1 previous similar message [ 3871.713450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3871.721688] Lustre: Skipped 1 previous similar message [ 3872.223526] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3872.483615] Lustre: server umount lustre-MDT0000 complete [ 3875.887849] LustreError: 134231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831669 with bad export cookie 2765947989301050140 [ 3875.892312] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3875.902290] LustreError: 134231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3876.215291] Lustre: server umount lustre-MDT0001 complete [ 3883.565550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3884.152177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3888.773756] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3889.642983] LustreError: 138424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3889.666032] LustreError: 138424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3893.733099] LustreError: 138423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3896.340904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3896.716130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3900.633522] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3906.907744] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3911.460521] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3916.919499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3917.167650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3922.573060] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3929.212211] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3933.031864] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3933.209751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3937.903661] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3938.100297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3946.355872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3946.477708] Lustre: lustre-OST0001: new disk, initializing [ 3946.480906] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3946.485932] Lustre: 141794:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3946.566264] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3952.082780] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 3955.249383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3955.260357] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3955.360765] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3959.586754] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3959.766358] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3963.831345] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3964.034224] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3968.811293] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d364aebc000.ost_server_uuid 50 [ 3970.084738] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d364aebc000.ost_server_uuid in FULL state after 0 sec [ 3981.282945] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3981.288683] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3981.296660] LustreError: Skipped 2 previous similar messages [ 3981.301571] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3981.312346] Lustre: Skipped 1 previous similar message [ 3981.328279] Lustre: Skipped 1 previous similar message [ 3983.441738] Lustre: server umount lustre-OST0001 complete [ 3986.409267] LustreError: 140245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3986.420631] LustreError: 140245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3987.940800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3987.956210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3989.984757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3990.006371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3990.014072] Lustre: Skipped 1 previous similar message [ 3991.532845] LustreError: 140244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.555226] LustreError: 140244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3996.641557] LustreError: 140865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3996.673542] LustreError: 140865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4001.247279] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4001.414334] Lustre: server umount lustre-OST0000 complete [ 4006.881153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4006.885702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4006.899703] Lustre: Skipped 2 previous similar messages [ 4006.907172] Lustre: Skipped 5 previous similar messages [ 4011.181583] Lustre: server umount lustre-MDT0000 complete [ 4012.005075] LustreError: 138423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4012.027620] LustreError: 138423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4015.077450] LustreError: 138403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831808 with bad export cookie 2765947989301051022 [ 4015.082855] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4015.092044] LustreError: 138403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4015.400636] Lustre: server umount lustre-MDT0001 complete [ 4034.044702] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 4036.811576] Key type lgssc unregistered [ 4037.120080] LNet: 144801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4037.126350] LNetError: 144801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4037.150379] LNet: Removed LNI 192.168.206.144@tcp [ 4038.155112] Key type .llcrypt unregistered [ 4038.156853] Key type ._llcrypt unregistered [ 4079.792286] Key type ._llcrypt registered [ 4079.795310] Key type .llcrypt registered [ 4079.897258] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 4093.122951] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 4094.338276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4094.353764] alg: No test for adler32 (adler32-zlib) [ 4095.387394] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 4095.649944] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 4097.311532] Key type lgssc registered [ 4098.405930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4132.697300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4132.731724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4134.029344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4134.064796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4134.119901] Lustre: lustre-MDT0000: new disk, initializing [ 4134.183317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4134.194642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4139.183191] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4148.842528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4149.000605] Lustre: 148938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4149.077602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4149.086734] Lustre: Skipped 1 previous similar message [ 4149.161883] Lustre: lustre-MDT0001: new disk, initializing [ 4149.262804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4149.299970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4149.308239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4152.754868] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4160.432184] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4164.200634] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4169.596278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4169.821269] Lustre: lustre-OST0000: new disk, initializing [ 4169.824709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4169.829545] Lustre: 150901:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4169.895689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4175.741602] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4179.006308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4179.028408] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4179.142772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4184.207993] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4188.944263] Lustre: Failing over lustre-OST0000 [ 4189.105926] Lustre: server umount lustre-OST0000 complete [ 4206.048445] Lustre: 147158:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831983/real 1787831983] req@00000000c30e8161 x1874677610871168/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787831999 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4206.081701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4206.279545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4206.538628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4206.561569] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4207.137997] Lustre: 147158:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831984/real 1787831984] req@00000000898f8d83 x1874677610872192/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787832000 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4207.154985] Lustre: 147158:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4207.389176] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4210.656630] Lustre: 147160:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831988/real 1787831988] req@00000000f6f1e70e x1874677610872960/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787832004 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4212.049861] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4215.694916] Lustre: lustre-OST0000: Recovery over after 0:08, of 3 clients 3 recovered and 0 were evicted. [ 4215.696504] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4216.090148] Lustre: Failing over lustre-MDT0000 [ 4216.480617] Lustre: server umount lustre-MDT0000 complete [ 4216.736132] Lustre: 147161:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831993/real 1787831993] req@00000000cc9bf8ec x1874677610874112/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787832009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4216.754196] Lustre: 147161:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4220.897965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4220.899815] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4220.918078] Lustre: Skipped 2 previous similar messages [ 4221.983158] Lustre: 147158:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831998/real 1787831998] req@00000000427c7aa3 x1874677610875264/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787832014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4222.721994] LustreError: 148943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4222.743585] LustreError: 148943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4226.019158] LustreError: 150908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4226.052873] LustreError: 150908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4227.836839] LustreError: 148945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4231.138400] LustreError: 151622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4231.151729] LustreError: 151622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4234.275285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4234.526657] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4234.834903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4234.880768] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4238.075587] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4239.086789] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4239.854955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4239.866752] Lustre: Skipped 1 previous similar message [ 4239.891568] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4239.950571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4246.497215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4246.502949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4246.519874] Lustre: Skipped 1 previous similar message [ 4246.526274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4250.089951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4250.096678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4252.368825] Lustre: server umount lustre-OST0000 complete [ 4260.321650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4260.326547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4260.348314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4260.354203] Lustre: Skipped 2 previous similar messages [ 4261.983718] Lustre: server umount lustre-MDT0000 complete [ 4265.206955] LustreError: 148930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832058 with bad export cookie 12193589403379989572 [ 4265.215064] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4265.217644] LustreError: 148930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4265.489209] Lustre: server umount lustre-MDT0001 complete [ 4274.435622] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 4276.690509] Key type lgssc unregistered [ 4276.906834] LNet: 154574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4276.914207] LNetError: 154574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4276.926862] LNet: Removed LNI 192.168.206.144@tcp [ 4277.680082] Key type .llcrypt unregistered [ 4277.682373] Key type ._llcrypt unregistered [ 4311.301486] Key type ._llcrypt registered [ 4311.304199] Key type .llcrypt registered [ 4311.442494] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 4312.610577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4312.636427] alg: No test for adler32 (adler32-zlib) [ 4313.811485] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 4314.132096] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 4315.839829] Key type lgssc registered [ 4317.065758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4324.114581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4324.130136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4325.512294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4329.653952] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4336.102986] LustreError: 156093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4336.121183] LustreError: 156093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4338.203536] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4341.217956] LustreError: 156094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4341.699489] LustreError: 156123:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4341.709287] LustreError: 156123:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4341.717202] LustreError: 156123:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4342.191586] Lustre: server umount lustre-MDT0000 complete [ 4355.902813] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 4365.081509] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 4397.389861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4397.559883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4397.600141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4397.679829] Lustre: lustre-MDT0000: new disk, initializing [ 4397.804162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4397.832805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4402.142928] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4412.163490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4412.284901] Lustre: 159425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4412.365853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4412.372514] Lustre: Skipped 1 previous similar message [ 4412.430641] Lustre: lustre-MDT0001: new disk, initializing [ 4412.541930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4412.583618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4412.592677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4416.804373] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4424.968337] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4430.105821] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4436.056804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4436.252996] Lustre: lustre-OST0000: new disk, initializing [ 4436.256412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4436.261676] Lustre: 161392:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4436.344598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4441.781655] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4444.755934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4444.762488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4444.836860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4450.212392] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4454.620195] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4456.503675] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 08:04:08 (1787832248) [ 4699.105478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4699.113715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4699.129761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4701.153720] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4701.156189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4701.171448] Lustre: Skipped 2 previous similar messages [ 4704.263301] Lustre: server umount lustre-MDT0000 complete [ 4706.276606] LustreError: 159431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4706.299065] LustreError: 159431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4707.823758] LustreError: 159416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832501 with bad export cookie 115877428069510459 [ 4707.827122] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4707.838332] LustreError: 159416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4711.391763] LustreError: 160986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4711.394748] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4711.433712] Lustre: Skipped 1 previous similar message [ 4711.442182] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4714.319287] Lustre: server umount lustre-MDT0001 complete [ 4732.384128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4732.705449] Lustre: server umount lustre-OST0000 complete [ 4744.655995] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 4755.214524] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 4794.800919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4795.140737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4795.203273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4795.359493] Lustre: lustre-MDT0000: new disk, initializing [ 4795.430807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4795.457568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4799.254162] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4809.382713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4809.521946] Lustre: 166057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4809.614473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4809.624490] Lustre: Skipped 1 previous similar message [ 4809.751490] Lustre: lustre-MDT0001: new disk, initializing [ 4809.843790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4809.907630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4809.922827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4814.236534] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4823.856452] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4829.576528] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4835.703039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4835.911339] Lustre: lustre-OST0000: new disk, initializing [ 4835.914325] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4835.919296] Lustre: 168026:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4836.029341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4842.745734] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4843.127962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4843.141473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4843.249178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4854.059497] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4858.434189] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4858.664675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4863.860967] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4864.063378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4869.097233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4869.098367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4869.107408] LustreError: Skipped 1 previous similar message [ 4869.108623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4869.118658] Lustre: Skipped 1 previous similar message [ 4869.121560] Lustre: Skipped 1 previous similar message [ 4874.218749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4874.226780] Lustre: Skipped 2 previous similar messages [ 4880.352204] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4880.508644] Lustre: server umount lustre-OST0000 complete [ 4884.453576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4884.463759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4884.477990] Lustre: Skipped 1 previous similar message [ 4884.495150] Lustre: Skipped 3 previous similar messages [ 4890.414418] Lustre: server umount lustre-MDT0000 complete [ 4893.812821] LustreError: 168025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832687 with bad export cookie 115877428069669289 [ 4893.817853] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4893.827558] LustreError: 168025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4894.173369] Lustre: server umount lustre-MDT0001 complete [ 4901.647061] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:11:33 (1787832693) [ 4916.880235] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 4929.711398] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 4961.206366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4961.422462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4961.477232] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4961.590052] Lustre: lustre-MDT0000: new disk, initializing [ 4961.677208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4961.695738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4965.381295] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4974.754814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4974.881677] Lustre: 172780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4975.103646] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4978.619141] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 4986.519603] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4991.873330] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4998.624156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4998.943331] Lustre: lustre-OST0000: new disk, initializing [ 4998.945830] Lustre: Skipped 1 previous similar message [ 4998.950564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4998.959975] Lustre: Skipped 2 previous similar messages [ 4998.967255] Lustre: 174758:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5004.951964] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5006.444954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5006.451745] Lustre: Skipped 1 previous similar message [ 5006.455410] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5006.570505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5013.931549] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5022.184429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5022.190610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5022.212704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5022.245988] Lustre: Skipped 3 previous similar messages [ 5026.958163] Lustre: server umount lustre-OST0000 complete [ 5031.392242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5037.442924] Lustre: server umount lustre-MDT0000 complete [ 5037.536604] LustreError: 172792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5037.558531] LustreError: 172792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5041.393973] LustreError: 172773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832835 with bad export cookie 115877428069670171 [ 5041.396782] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5041.404755] LustreError: 172773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5041.871980] Lustre: server umount lustre-MDT0001 complete [ 5050.502410] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:14:02 (1787832842) [ 5065.168463] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 5074.858194] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 5108.698342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5108.935147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5108.960898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5109.022270] Lustre: lustre-MDT0000: new disk, initializing [ 5109.086033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5109.090247] Lustre: Skipped 2 previous similar messages [ 5109.110696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5113.208845] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5123.810135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5124.031518] Lustre: 179081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5124.440767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5128.926756] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5137.180521] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5141.518609] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5147.887280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5148.217934] Lustre: 181061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5153.637501] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5156.563863] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5162.010421] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5171.688169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5171.697873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5171.713443] Lustre: Skipped 3 previous similar messages [ 5171.720541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5171.723195] Lustre: Skipped 3 previous similar messages [ 5175.460858] Lustre: server umount lustre-OST0000 complete [ 5180.898273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5180.908671] LustreError: Skipped 1 previous similar message [ 5185.240968] Lustre: server umount lustre-MDT0000 complete [ 5187.552393] LustreError: 179087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5187.565835] LustreError: 179087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5188.439225] LustreError: 179072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832982 with bad export cookie 115877428069671256 [ 5188.443393] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5188.452023] LustreError: 179072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5188.786125] Lustre: server umount lustre-MDT0001 complete [ 5198.012683] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 5200.480412] Key type lgssc unregistered [ 5200.805960] LNet: 182917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5200.819514] LNetError: 182917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5200.841651] LNet: Removed LNI 192.168.206.144@tcp [ 5201.526680] Key type .llcrypt unregistered [ 5201.528740] Key type ._llcrypt unregistered [ 5224.798570] Key type ._llcrypt registered [ 5224.801959] Key type .llcrypt registered [ 5224.876615] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 5226.380081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5226.406836] alg: No test for adler32 (adler32-zlib) [ 5227.500428] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 5227.777521] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 5229.415223] Key type lgssc registered [ 5230.534961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5237.437677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5237.465165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5238.754134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5243.278423] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5248.993802] LustreError: 184334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5249.013403] LustreError: 184334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5251.595524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5252.028534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5255.949332] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5263.348407] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5268.693023] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5274.320413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5274.573103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5281.025642] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5283.824763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5288.295200] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5307.360926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5307.374165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5307.395936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5309.413879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5309.420358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5309.428209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5312.731969] Lustre: server umount lustre-OST0000 complete [ 5318.624863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5318.639808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5318.660960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5318.669129] Lustre: Skipped 1 previous similar message [ 5323.361344] Lustre: server umount lustre-MDT0000 complete [ 5324.769489] LustreError: 186550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5324.791908] LustreError: 186550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5327.735023] LustreError: 184313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833121 with bad export cookie 249940423394267694 [ 5327.751884] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5328.074610] Lustre: server umount lustre-MDT0001 complete [ 5339.623944] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 5342.302714] Key type lgssc unregistered [ 5342.561438] LNet: 187892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5342.574823] LNetError: 187892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5342.615581] LNet: Removed LNI 192.168.206.144@tcp [ 5343.443798] Key type .llcrypt unregistered [ 5343.448221] Key type ._llcrypt unregistered [ 5366.661748] Key type ._llcrypt registered [ 5366.664023] Key type .llcrypt registered [ 5366.759375] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 5367.718986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5367.873586] alg: No test for adler32 (adler32-zlib) [ 5368.892542] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 5369.083223] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 5370.719297] Key type lgssc registered [ 5371.875775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5378.728646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5378.750627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5380.164403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5383.812455] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5389.295111] LustreError: 189319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5389.330899] LustreError: 189319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5391.276497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5391.578510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5395.665596] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5401.675192] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5406.720577] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5412.480031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5412.736725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5418.749560] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5425.023386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5425.168759] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5432.288368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5432.297537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5432.313593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5433.132498] Lustre: lustre-OST0000: Not available for connect from 192.168.206.44@tcp (stopping) [ 5435.366322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5435.371140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5435.373570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5437.418883] Lustre: server umount lustre-OST0000 complete [ 5445.499345] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d3649a7d800.ost_server_uuid 50 [ 5446.886816] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5460.965550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5460.970915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5460.980889] Lustre: Skipped 1 previous similar message [ 5461.001125] Lustre: Skipped 2 previous similar messages [ 5466.819805] Lustre: server umount lustre-MDT0000 complete [ 5467.106327] LustreError: 189316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5467.122152] LustreError: 189316:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5470.161250] LustreError: 189299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833263 with bad export cookie 8405019626753040552 [ 5470.166479] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5470.173945] LustreError: 189299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5470.521947] Lustre: server umount lustre-MDT0001 complete [ 5477.781101] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 08:21:09 (1787833269) [ 5483.098702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5483.460185] LustreError: 193237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5483.542703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5487.820296] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5488.609735] LustreError: 193238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5493.731632] LustreError: 193237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5495.949542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5496.360641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5500.622624] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5508.211370] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5513.836059] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5520.600313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5521.028836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5528.154712] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5529.212181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5535.227152] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5541.602367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5541.815743] Lustre: lustre-OST0001: new disk, initializing [ 5541.821072] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5541.827238] Lustre: 196079:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5541.930424] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5547.573388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5547.585415] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5547.706338] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5547.740563] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5556.170589] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5560.289586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5560.297535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5560.323580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5560.328272] Lustre: Skipped 2 previous similar messages [ 5563.365124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5564.418645] Lustre: server umount lustre-OST0000 complete [ 5567.234234] LustreError: 195058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5570.336511] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5570.529437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5575.096737] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5575.325182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5583.846712] LustreError: 196087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.873376] LustreError: 196087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 5588.967940] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5588.969296] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5588.979163] LustreError: Skipped 1 previous similar message [ 5588.979886] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5588.979891] Lustre: Skipped 3 previous similar messages [ 5589.018719] Lustre: Skipped 2 previous similar messages [ 5594.567432] Lustre: server umount lustre-OST0001 complete [ 5598.689462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5598.700398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5604.024424] Lustre: server umount lustre-MDT0000 complete [ 5607.775938] LustreError: 193217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833401 with bad export cookie 8405019626753041623 [ 5607.796018] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5607.802272] LustreError: 193217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5608.266338] Lustre: server umount lustre-MDT0001 complete [ 5628.238516] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 08:23:40 (1787833420) [ 5634.035476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5634.348470] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5634.422614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5634.643611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5638.481103] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5646.708907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5646.798219] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5646.823684] Lustre: 199516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5651.174881] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5658.311308] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5663.629318] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5670.215098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5670.354123] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5676.369987] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5677.194631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5684.203469] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5689.561366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5689.695134] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5695.581157] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5701.539561] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5707.059239] Lustre: lustre-OST0000: Not available for connect from 192.168.206.44@tcp (stopping) [ 5707.067022] Lustre: Skipped 6 previous similar messages [ 5707.747354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5707.759382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5707.774030] Lustre: Skipped 1 previous similar message [ 5717.985230] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5718.130780] Lustre: server umount lustre-OST0000 complete [ 5718.499250] LustreError: 201781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5718.523927] LustreError: 201781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 5738.988932] Lustre: server umount lustre-OST0001 complete [ 5744.097651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5744.099301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5744.116485] Lustre: Skipped 4 previous similar messages [ 5744.129382] Lustre: Skipped 10 previous similar messages [ 5756.383570] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5756.591099] Lustre: server umount lustre-MDT0000 complete [ 5760.472515] LustreError: 202401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833554 with bad export cookie 8405019626753043030 [ 5760.485531] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5760.486051] LustreError: 202401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5760.803305] Lustre: server umount lustre-MDT0001 complete [ 5781.516232] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 08:26:13 (1787833573) [ 5794.971879] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 5803.744308] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 5837.449147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5837.690640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5837.712058] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5837.760759] Lustre: lustre-MDT0000: new disk, initializing [ 5837.817167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5837.822907] Lustre: Skipped 3 previous similar messages [ 5837.834930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5841.592616] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5851.339124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5851.452114] Lustre: 207613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5851.511266] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5851.521394] Lustre: Skipped 1 previous similar message [ 5851.616171] Lustre: lustre-MDT0001: new disk, initializing [ 5851.753435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5851.760672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5855.716509] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5864.895858] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5870.695427] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5876.638715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5876.861858] Lustre: lustre-OST0000: new disk, initializing [ 5876.867619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5876.880789] Lustre: 209582:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5883.418482] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5883.457903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5883.471844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5883.602315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5893.301426] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5897.894978] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5898.119543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5902.691721] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5902.980089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5909.472994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5909.473447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5909.482039] LustreError: Skipped 4 previous similar messages [ 5909.485028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5909.495142] Lustre: Skipped 1 previous similar message [ 5909.511201] Lustre: Skipped 5 previous similar messages [ 5911.037071] Lustre: server umount lustre-OST0000 complete [ 5920.980635] Lustre: server umount lustre-MDT0000 complete [ 5924.535944] LustreError: 207608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833718 with bad export cookie 8405019626753044598 [ 5924.545781] LustreError: 207608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5924.550181] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5924.866463] Lustre: server umount lustre-MDT0001 complete [ 5933.078691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5933.558219] LustreError: 211800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5933.574823] LustreError: 211800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 5938.266844] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5945.985214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5950.757425] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5958.510568] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5964.460693] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5970.933285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5971.217309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5971.225082] Lustre: Skipped 4 previous similar messages [ 5978.583065] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 5987.009824] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5991.602829] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5991.839304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5996.744280] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5996.947916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6001.131930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6005.397820] Lustre: server umount lustre-OST0000 complete [ 6011.779654] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6016.377742] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6045.506027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6052.631571] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6060.644721] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6063.528904] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6065.387082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.396850] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.403891] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6065.417111] Lustre: Skipped 1 previous similar message [ 6070.754349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6070.787258] Lustre: Skipped 4 previous similar messages [ 6070.805921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6070.813228] Lustre: Skipped 7 previous similar messages [ 6074.472263] Lustre: server umount lustre-OST0000 complete [ 6092.255793] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6092.438842] Lustre: server umount lustre-MDT0000 complete [ 6095.723964] LustreError: 211781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833889 with bad export cookie 8405019626753045480 [ 6095.731375] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6095.739300] LustreError: 211781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6096.035748] Lustre: server umount lustre-MDT0001 complete [ 6103.641231] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 08:31:35 (1787833895) [ 6109.643394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6110.165745] LustreError: 217767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6110.183436] LustreError: 217767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6114.430888] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6122.127444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6126.815728] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6133.906291] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6138.839188] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6145.192109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6151.452831] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6158.919955] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6163.212665] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6163.495460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6168.921142] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6169.329226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6177.557613] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6177.785448] Lustre: lustre-OST0001: new disk, initializing [ 6177.788837] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6177.794396] Lustre: 221133:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6184.074832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6184.087492] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6184.208541] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6184.782646] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6194.975878] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6199.652646] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6199.835926] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6204.250181] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6204.487097] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6210.031837] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6210.052249] LustreError: Skipped 4 previous similar messages [ 6212.594611] Lustre: server umount lustre-OST0001 complete [ 6218.965990] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6219.294563] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6223.650747] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6223.861579] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6252.644246] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6252.888761] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6252.896988] Lustre: Skipped 5 previous similar messages [ 6258.166717] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6258.197494] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6258.201503] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6259.461192] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6266.538985] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6268.065336] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6275.739453] Lustre: server umount lustre-OST0001 complete [ 6298.444384] Lustre: server umount lustre-OST0000 complete [ 6308.565547] Lustre: server umount lustre-MDT0000 complete [ 6312.228485] LustreError: 217749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834105 with bad export cookie 8405019626753046446 [ 6312.232791] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6312.241506] LustreError: 217749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6318.766347] Lustre: server umount lustre-MDT0001 complete [ 6341.258694] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 08:35:32 (1787834132) [ 6347.704091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6348.007443] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6348.034289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6352.110438] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6360.063256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.167801] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6360.190475] Lustre: 226380:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6364.787730] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6371.324751] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6376.901898] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6383.440855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6383.591148] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6389.398135] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6396.275720] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6404.800112] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6404.880825] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6411.239195] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6417.682299] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6419.045518] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6423.469671] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6423.687853] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6428.308594] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6428.486723] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6433.641815] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d3651e7f800.ost_server_uuid 50 [ 6435.077413] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d3651e7f800.ost_server_uuid in FULL state after 0 sec [ 6436.096619] Lustre: Permanently deactivating lustre-OST0001 [ 6436.107361] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6439.879665] Lustre: Permanently reactivating lustre-OST0001 [ 6446.057389] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6446.091241] Lustre: Skipped 11 previous similar messages [ 6446.099137] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6446.112147] Lustre: Skipped 19 previous similar messages [ 6449.330929] Lustre: server umount lustre-OST0001 complete [ 6451.686228] LustreError: 228223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6451.708921] LustreError: 228223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 36 previous similar messages [ 6459.096298] Lustre: server umount lustre-OST0000 complete [ 6468.822796] Lustre: server umount lustre-MDT0000 complete [ 6472.294733] LustreError: 226371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834266 with bad export cookie 8405019626753047734 [ 6472.303290] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6472.310701] LustreError: 226371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6472.752908] Lustre: server umount lustre-MDT0001 complete [ 6494.231933] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 08:38:06 (1787834286) [ 6501.268381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6501.823550] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6501.863776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6501.873739] Lustre: Skipped 5 previous similar messages [ 6507.294248] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6515.718871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6516.021140] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6516.082341] Lustre: 233071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6522.106953] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6530.247523] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6536.648813] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6544.438326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6544.717785] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6544.740705] Lustre: Permanently deactivating lustre-OST0000 [ 6544.764654] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6552.078218] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6552.612227] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6552.725435] LustreError: 235327:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 6559.678839] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6565.444122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6565.545459] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6571.459749] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6573.190349] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6577.322562] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6578.752827] Lustre: Permanently reactivating lustre-OST0000 [ 6578.775538] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6578.780169] Lustre: Skipped 2 previous similar messages [ 6582.445374] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6582.455534] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6582.475722] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6582.483952] Lustre: Skipped 1 previous similar message [ 6582.512780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6582.534580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6593.505262] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6593.511297] LustreError: Skipped 6 previous similar messages [ 6594.667974] Lustre: server umount lustre-OST0001 complete [ 6604.163319] Lustre: server umount lustre-OST0000 complete [ 6614.047040] Lustre: server umount lustre-MDT0000 complete [ 6617.721545] LustreError: 233062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834411 with bad export cookie 8405019626753049414 [ 6617.730843] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6617.741402] LustreError: 233062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6618.290870] Lustre: server umount lustre-MDT0001 complete [ 6626.985022] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 6629.423378] Key type lgssc unregistered [ 6629.676856] LNet: 237952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6629.685710] LNetError: 237952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6629.703350] LNet: Removed LNI 192.168.206.144@tcp [ 6630.643355] Key type .llcrypt unregistered [ 6630.645667] Key type ._llcrypt unregistered [ 6653.180945] Key type ._llcrypt registered [ 6653.183934] Key type .llcrypt registered [ 6653.336679] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 6654.551105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6654.583085] alg: No test for adler32 (adler32-zlib) [ 6655.734174] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 6656.104519] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 6657.905278] Key type lgssc registered [ 6659.181936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6671.544924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6671.569646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6673.271791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6676.746560] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6682.594894] LustreError: 239421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6682.610851] LustreError: 239421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6684.680395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6684.779925] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6684.802307] Lustre: 239412:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6684.817680] Lustre: Permanently deactivating lustre-MDT0001 [ 6684.832823] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6685.140267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6686.997300] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6687.005805] LustreError: 239452:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6687.026318] LustreError: 239452:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6687.038632] LustreError: 239452:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 1, failed: rc = -108 [ 6689.194630] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6696.032067] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6700.585504] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6706.202890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6706.552839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6709.616564] LustreError: 241263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.644675] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 6711.737372] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6714.852249] LustreError: 241264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6717.932209] LustreError: 241263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6717.952927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6717.964257] LustreError: 241263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6718.592221] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6723.041657] LustreError: 241263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6723.070752] LustreError: 241263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6725.179239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6725.323354] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6730.739921] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6730.762048] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6731.428882] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6738.352277] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6740.629937] Lustre: Permanently reactivating lustre-MDT0001 [ 6740.646354] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6740.650977] Lustre: Skipped 2 previous similar messages [ 6744.609792] Lustre: lustre-MDT0001: Client a823082c-54c0-4764-8a7c-e07ce569104a (at 192.168.206.44@tcp) reconnecting [ 6749.453554] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6750.657600] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6755.290990] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6755.461569] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6759.310772] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6759.507995] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6761.607277] Lustre: Permanently deactivating lustre-MDT0001 [ 6761.624711] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6761.628985] Lustre: Skipped 2 previous similar messages [ 6771.793396] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6781.922455] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6781.926519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6781.940596] Lustre: Skipped 1 previous similar message [ 6781.953526] Lustre: Skipped 3 previous similar messages [ 6784.614653] Lustre: server umount lustre-MDT0000 complete [ 6787.047665] LustreError: 239421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6787.068293] LustreError: 239421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6788.166510] LustreError: 242969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834581 with bad export cookie 1441556714875159334 [ 6788.168252] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6788.173516] LustreError: 242969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6788.580318] Lustre: server umount lustre-MDT0001 complete [ 6794.870645] Lustre: server umount lustre-OST0000 complete [ 6805.050309] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 08:43:17 (1787834597) [ 6814.147329] Lustre: server umount lustre-OST0001 complete [ 6820.391801] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 6829.037364] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 6863.240338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6863.449210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6863.461151] Lustre: Skipped 2 previous similar messages [ 6863.485097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6863.590360] Lustre: lustre-MDT0000: new disk, initializing [ 6863.655641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6863.670315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6868.339521] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6878.367877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6878.490789] Lustre: 247722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6878.553326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6878.560469] Lustre: Skipped 1 previous similar message [ 6878.643448] Lustre: lustre-MDT0001: new disk, initializing [ 6878.754041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6878.812158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6878.824995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6882.803348] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6891.107326] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6896.276935] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6902.743212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6902.961113] Lustre: lustre-OST0000: new disk, initializing [ 6902.964509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6902.969852] Lustre: 249691:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6903.070093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6908.364314] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6911.078045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6911.087548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6911.180534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6916.114178] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6917.853490] LustreError: 249698:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6926.667606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6926.833712] Lustre: lustre-OST0001: new disk, initializing [ 6926.839608] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6926.849512] Lustre: 250804:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6927.935121] LustreError: 249698:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6927.956383] LustreError: 247731:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6932.072763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6932.092712] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6932.138708] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6932.500958] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 6938.042645] LustreError: 247731:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6941.851474] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6943.455362] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6946.087323] Lustre: lustre-OST0001: Not available for connect from 192.168.206.44@tcp (stopping) [ 6947.810137] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6947.816625] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6947.822063] LustreError: Skipped 1 previous similar message [ 6947.837085] Lustre: Skipped 2 previous similar messages [ 6947.839525] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6947.856917] Lustre: Skipped 1 previous similar message [ 6951.171270] Lustre: lustre-OST0001: Not available for connect from 192.168.206.44@tcp (stopping) [ 6951.684897] Lustre: server umount lustre-OST0001 complete [ 6952.935358] LustreError: 249673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6952.955111] LustreError: 249673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6972.897274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6972.898775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6972.906747] LustreError: Skipped 1 previous similar message [ 6972.916795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6972.937896] Lustre: Skipped 2 previous similar messages [ 6972.957253] Lustre: Skipped 1 previous similar message [ 6983.135130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6983.329944] Lustre: server umount lustre-OST0000 complete [ 6988.772410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.777744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6988.785464] Lustre: Skipped 1 previous similar message [ 6988.799252] Lustre: Skipped 4 previous similar messages [ 6993.890628] LustreError: 247735:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6993.924394] LustreError: 247735:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6994.949775] Lustre: server umount lustre-MDT0000 complete [ 6998.743540] LustreError: 249690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834792 with bad export cookie 1441556714875162372 [ 6998.751405] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6998.760711] LustreError: 249690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7013.344421] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7013.654545] Lustre: server umount lustre-MDT0001 complete [ 7025.487373] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 7028.736357] Key type lgssc unregistered [ 7029.013625] LNet: 252846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7029.027839] LNetError: 252846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7029.056063] LNet: Removed LNI 192.168.206.144@tcp [ 7030.485552] Key type .llcrypt unregistered [ 7030.490034] Key type ._llcrypt unregistered [ 7066.695857] Key type ._llcrypt registered [ 7066.698165] Key type .llcrypt registered [ 7066.808757] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 7068.068440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7068.098043] alg: No test for adler32 (adler32-zlib) [ 7069.158951] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 7069.356857] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 7071.023157] Key type lgssc registered [ 7071.897620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7078.307562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7078.332945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7079.600653] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7079.625663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7079.781614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7083.968157] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7091.596630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7091.717472] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7091.746615] Lustre: 255038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7091.949379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7095.288734] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7101.441271] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7105.830161] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7111.048305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7111.270094] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7111.438858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7116.157566] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7120.055843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7122.470858] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7171.040723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7171.052645] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7171.082789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7171.557636] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7171.573559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7171.584574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7176.675592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7176.683629] Lustre: Skipped 1 previous similar message [ 7177.213694] Lustre: server umount lustre-OST0000 complete [ 7180.929703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7192.871444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7193.183618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7198.664304] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7201.386670] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7201.400823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7201.406404] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7201.426704] Lustre: Skipped 1 previous similar message [ 7205.560283] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7247.845415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7247.848148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7247.855292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7247.862150] Lustre: Skipped 1 previous similar message [ 7247.878899] Lustre: Skipped 1 previous similar message [ 7250.792812] Lustre: server umount lustre-OST0000 complete [ 7256.032179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7256.037135] LustreError: Skipped 1 previous similar message [ 7256.038714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7256.044985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7260.601122] Lustre: server umount lustre-MDT0000 complete [ 7263.629108] LustreError: 255029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835057 with bad export cookie 3159867999619538241 [ 7263.629921] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7263.642605] LustreError: 255029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7263.955676] Lustre: server umount lustre-MDT0001 complete [ 7274.112649] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 7276.681891] Key type lgssc unregistered [ 7276.895701] LNet: 260461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7276.907053] LNetError: 260461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7277.928721] LNet: Removed LNI 192.168.206.144@tcp [ 7278.741787] Key type .llcrypt unregistered [ 7278.744212] Key type ._llcrypt unregistered [ 7300.511715] Key type ._llcrypt registered [ 7300.514102] Key type .llcrypt registered [ 7300.636864] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 7302.208514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7302.250383] alg: No test for adler32 (adler32-zlib) [ 7303.285204] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 7303.478994] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 7305.151141] Key type lgssc registered [ 7306.288591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7314.093499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7314.114528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7315.806848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7320.673219] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7326.192271] LustreError: 261880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7326.215491] LustreError: 261880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7330.325396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7330.796410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7335.283615] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7343.169198] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7348.397649] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7357.084637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7357.631402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7360.685836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7367.279706] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7376.656951] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7401.953865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7401.955055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7401.965116] LustreError: Skipped 1 previous similar message [ 7401.966562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7402.002642] Lustre: Skipped 1 previous similar message [ 7405.026078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7405.041463] Lustre: Skipped 1 previous similar message [ 7407.154589] Lustre: server umount lustre-OST0000 complete [ 7410.144345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7410.162558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7410.167056] Lustre: Skipped 2 previous similar messages [ 7415.265548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7415.271347] Lustre: Skipped 1 previous similar message [ 7416.542213] Lustre: server umount lustre-MDT0000 complete [ 7419.812879] LustreError: 261861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835213 with bad export cookie 639912285908223759 [ 7419.816213] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7419.822239] LustreError: 261861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7420.144791] Lustre: server umount lustre-MDT0001 complete [ 7430.574230] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 7433.092865] Key type lgssc unregistered [ 7433.351895] LNet: 266193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7433.360132] LNetError: 266193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7433.376023] LNet: Removed LNI 192.168.206.144@tcp [ 7434.041707] Key type .llcrypt unregistered [ 7434.046856] Key type ._llcrypt unregistered [ 7448.112901] Key type ._llcrypt registered [ 7448.118115] Key type .llcrypt registered [ 7448.214899] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 7448.875455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7448.889601] alg: No test for adler32 (adler32-zlib) [ 7449.960666] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 7450.139045] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 7451.767727] Key type lgssc registered [ 7452.773478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7459.427122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7459.456755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7460.932709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7464.609723] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7471.072939] LustreError: 267410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7471.090561] LustreError: 267410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7471.189744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7471.479182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7475.228847] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7481.605146] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7486.126473] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7491.734481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7492.012718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7497.193601] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7504.033330] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7505.335886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7520.744983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.747972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7520.754756] Lustre: Skipped 1 previous similar message [ 7520.756144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7524.724298] Lustre: server umount lustre-OST0000 complete [ 7527.906456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7527.914651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7527.927657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7527.932585] Lustre: Skipped 1 previous similar message [ 7530.977737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7530.977784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7530.994691] Lustre: Skipped 1 previous similar message [ 7536.097753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7536.103749] Lustre: Skipped 1 previous similar message [ 7541.217831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7541.224458] Lustre: Skipped 1 previous similar message [ 7542.239139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7542.324325] Lustre: server umount lustre-MDT0000 complete [ 7545.274693] LustreError: 267389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835339 with bad export cookie 9785651710966693120 [ 7545.284642] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7545.285508] LustreError: 267389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7545.476959] Lustre: server umount lustre-MDT0001 complete [ 7555.815390] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 7559.068712] Key type lgssc unregistered [ 7559.530863] LNet: 271093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7559.542685] LNetError: 271093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7559.565759] LNet: Removed LNI 192.168.206.144@tcp [ 7560.732280] Key type .llcrypt unregistered [ 7560.737106] Key type ._llcrypt unregistered [ 7589.402540] Key type ._llcrypt registered [ 7589.405500] Key type .llcrypt registered [ 7589.486647] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 7590.389871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7590.419193] alg: No test for adler32 (adler32-zlib) [ 7591.499820] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 7591.700698] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 7593.335420] Key type lgssc registered [ 7594.240371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7600.568470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7600.578859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7601.977814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7606.349749] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7611.362165] LustreError: 272509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7611.410114] LustreError: 272509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7615.012989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7615.764462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7620.844588] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7630.391958] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7637.705846] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7645.476460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7645.902254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7646.908984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7651.663770] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7658.573747] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7682.023829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7682.025453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7682.032128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7682.047731] Lustre: Skipped 1 previous similar message [ 7686.199574] Lustre: server umount lustre-OST0000 complete [ 7692.262552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7692.282426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7692.291464] Lustre: Skipped 2 previous similar messages [ 7697.379847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7697.389782] Lustre: Skipped 1 previous similar message [ 7701.472630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7704.031156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7704.219813] Lustre: server umount lustre-MDT0000 complete [ 7706.595284] LustreError: 272509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7706.620954] LustreError: 272509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7707.534299] LustreError: 272489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835501 with bad export cookie 9703867879170493316 [ 7707.542498] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7707.545158] LustreError: 272489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7707.763338] Lustre: server umount lustre-MDT0001 complete [ 7715.749811] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 7717.964602] Key type lgssc unregistered [ 7718.208581] LNet: 276819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7718.220496] LNetError: 276819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7719.268748] LNet: Removed LNI 192.168.206.144@tcp [ 7719.979893] Key type .llcrypt unregistered [ 7719.994272] Key type ._llcrypt unregistered [ 7735.390610] Key type ._llcrypt registered [ 7735.393377] Key type .llcrypt registered [ 7735.488939] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 7736.382626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7736.401153] alg: No test for adler32 (adler32-zlib) [ 7737.389166] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 7737.625988] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 7739.263152] Key type lgssc registered [ 7740.364994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7747.034553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7747.048762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7748.338186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7751.974627] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7758.741419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7758.819788] LustreError: 278054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7758.831697] LustreError: 278054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7759.113978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7763.072832] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7769.666733] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7775.368901] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7782.150386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7782.595442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7788.737824] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7793.855956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7797.048198] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7809.510177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7809.515980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7809.522137] LustreError: Skipped 1 previous similar message [ 7809.524882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7809.539065] Lustre: Skipped 1 previous similar message [ 7811.095478] Lustre: server umount lustre-OST0000 complete [ 7815.648832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7815.661164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7815.673251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7815.676932] Lustre: Skipped 1 previous similar message [ 7819.745191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7819.760869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7819.771319] Lustre: Skipped 1 previous similar message [ 7821.038095] Lustre: server umount lustre-MDT0000 complete [ 7824.864470] LustreError: 278061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7824.897184] LustreError: 278061:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7825.314593] LustreError: 278021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835619 with bad export cookie 14752440242036196847 [ 7825.320118] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7825.328955] LustreError: 278021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7825.704429] Lustre: server umount lustre-MDT0001 complete [ 7836.707486] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 7839.336650] Key type lgssc unregistered [ 7839.623909] LNet: 281757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7839.630144] LNetError: 281757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7840.679354] LNet: Removed LNI 192.168.206.144@tcp [ 7841.553563] Key type .llcrypt unregistered [ 7841.555648] Key type ._llcrypt unregistered [ 7854.901764] Key type ._llcrypt registered [ 7854.903379] Key type .llcrypt registered [ 7855.070262] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7873.358849] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 7889.435629] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 7890.535556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7890.772824] alg: No test for adler32 (adler32-zlib) [ 7891.834915] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 7892.078353] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 7893.737224] Key type lgssc registered [ 7895.089746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7931.217970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7931.281953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7932.647248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7932.703837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7932.803715] Lustre: lustre-MDT0000: new disk, initializing [ 7932.924468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7932.958086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7939.825273] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7951.893745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7951.988780] Lustre: 285321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7952.032231] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7952.036507] Lustre: Skipped 1 previous similar message [ 7952.091778] Lustre: lustre-MDT0001: new disk, initializing [ 7952.170099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7952.212705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7952.221299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7955.782404] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7963.635410] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7968.498764] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7974.325087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7974.617450] Lustre: lustre-OST0000: new disk, initializing [ 7974.622926] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7974.627891] Lustre: 287290:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7974.741464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7981.038434] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 7984.229709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7984.242235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7984.329428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7991.390806] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7996.964621] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7997.235881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8003.080522] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8003.322645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8010.208068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8010.214902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8010.222918] LustreError: Skipped 1 previous similar message [ 8010.225891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8010.277021] Lustre: Skipped 1 previous similar message [ 8011.502703] Lustre: server umount lustre-OST0000 complete [ 8015.333206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8015.352147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8015.361577] Lustre: Skipped 2 previous similar messages [ 8020.448970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8020.454251] Lustre: Skipped 1 previous similar message [ 8021.543370] Lustre: server umount lustre-MDT0000 complete [ 8025.165705] LustreError: 285313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835818 with bad export cookie 2047335392279330680 [ 8025.171136] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8025.178316] LustreError: 285313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8025.421815] Lustre: server umount lustre-MDT0001 complete [ 8033.396667] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 09:03:45 (1787835825) [ 8040.042515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8040.777572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8045.099188] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8046.051222] LustreError: 289700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8046.092689] LustreError: 289700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8051.172119] LustreError: 289701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8055.075438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8055.429865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8061.250613] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8069.837787] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8075.081217] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8081.637781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8081.952237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8087.782739] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8093.993450] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8110.560732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8110.565810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8110.583645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8110.599600] Lustre: Skipped 2 previous similar messages [ 8112.883594] Lustre: server umount lustre-OST0000 complete [ 8116.704579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8116.711735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8116.731865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8116.736247] Lustre: Skipped 1 previous similar message [ 8120.804786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8122.311232] Lustre: server umount lustre-MDT0000 complete [ 8125.852350] LustreError: 289682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835919 with bad export cookie 2047335392279331562 [ 8125.859351] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8125.864094] LustreError: 289682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8126.314972] Lustre: server umount lustre-MDT0001 complete [ 8136.883231] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 8139.932813] Key type lgssc unregistered [ 8140.328766] LNet: 293260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8140.337382] LNetError: 293260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8140.386190] LNet: Removed LNI 192.168.206.144@tcp [ 8141.329133] Key type .llcrypt unregistered [ 8141.331479] Key type ._llcrypt unregistered [ 8165.170989] Key type ._llcrypt registered [ 8165.173910] Key type .llcrypt registered [ 8165.295427] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 8181.940521] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8183.422885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8183.511544] alg: No test for adler32 (adler32-zlib) [ 8184.669422] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 8185.051775] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 8186.751195] Key type lgssc registered [ 8188.481133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8237.849958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8237.892387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8239.108790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8239.127412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8239.181805] Lustre: lustre-MDT0000: new disk, initializing [ 8239.241287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8239.257168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8243.442931] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8253.717831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8253.833300] Lustre: 297103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8253.893886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8253.898133] Lustre: Skipped 1 previous similar message [ 8253.954725] Lustre: lustre-MDT0001: new disk, initializing [ 8254.049534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8254.093508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8254.108744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8257.941519] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8267.631260] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8274.821719] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8281.811514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8282.236514] Lustre: lustre-OST2710: new disk, initializing [ 8282.243098] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8282.253198] Lustre: 299074:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8282.371593] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8287.814899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8287.829793] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8287.992752] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8289.321951] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8298.322145] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8304.951281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8305.089778] Lustre: lustre-OST03e8: new disk, initializing [ 8305.092574] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8305.097794] Lustre: 300137:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8305.172689] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8310.896066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8310.912552] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8310.920817] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8311.031697] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8318.205206] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8326.397661] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8326.693570] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8331.576361] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8331.749511] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8336.663216] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8336.885400] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8343.211556] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8343.528427] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8351.201075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8351.231516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8351.269177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8352.243622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8352.246665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8352.257734] Lustre: Skipped 1 previous similar message [ 8352.305039] Lustre: Skipped 2 previous similar messages [ 8356.063977] Lustre: server umount lustre-MDT0000 complete [ 8361.616854] LustreError: 297093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836155 with bad export cookie 9234788461075993929 [ 8361.620268] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8361.632543] LustreError: 297093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8362.222708] Lustre: server umount lustre-MDT0001 complete [ 8377.724464] Lustre: server umount lustre-OST2710 complete [ 8378.591113] Lustre: 294899:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787836156/real 1787836156] req@000000001630a4f7 x1874681898537344/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787836172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8378.620986] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8382.147760] Lustre: server umount lustre-OST03e8 complete [ 8404.275866] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 8416.575513] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8452.773292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8452.997994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8453.021457] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8453.135429] Lustre: lustre-MDT0000: new disk, initializing [ 8453.273217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8453.284858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8457.449538] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8466.211490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8466.282291] Lustre: 305392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8466.321154] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8466.324798] Lustre: Skipped 1 previous similar message [ 8466.397360] Lustre: lustre-MDT0001: new disk, initializing [ 8466.496735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8466.541051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8466.550479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8470.322435] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8478.059202] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8482.588688] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8488.324105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8488.642789] Lustre: lustre-OST0000: new disk, initializing [ 8488.649443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8488.658858] Lustre: 307356:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8488.738809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8494.079915] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8495.669874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8495.687412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8495.785073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8502.548769] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8506.591578] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8506.785428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8510.738758] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8510.932938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8516.583038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8516.586079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8516.589867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8516.589875] Lustre: Skipped 2 previous similar messages [ 8516.591595] Lustre: Skipped 1 previous similar message [ 8518.798475] Lustre: server umount lustre-OST0000 complete [ 8522.720821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8522.723163] LustreError: Skipped 1 previous similar message [ 8522.728190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8522.743324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8522.746044] Lustre: Skipped 1 previous similar message [ 8526.818788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8526.826469] Lustre: Skipped 1 previous similar message [ 8528.491360] Lustre: server umount lustre-MDT0000 complete [ 8531.489148] LustreError: 305382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836325 with bad export cookie 9234788461075995623 [ 8531.490693] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8531.502551] LustreError: 305382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8531.757455] Lustre: server umount lustre-MDT0001 complete [ 8539.230985] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 09:12:11 (1787836331) [ 8540.416665] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8542.016518] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 09:12:13 (1787836333) [ 8545.542724] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8570.724496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8571.056975] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8571.082935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8571.212497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8575.752977] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8584.026708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8584.078796] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8584.094370] Lustre: 311216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8589.031770] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8613.996792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8614.283977] Lustre: Denying initial registration attempt from nid 192.168.206.144@tcp, specified as failover [ 8614.293083] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8614.300745] LustreError: 312669:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8614.308682] LustreError: 312669:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8614.314066] LustreError: 312669:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8614.407487] Lustre: server umount lustre-OST0000 complete [ 8614.411362] LustreError: 312669:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8623.595416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8623.598311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8623.612651] Lustre: Skipped 2 previous similar messages [ 8623.632972] Lustre: Skipped 1 previous similar message [ 8627.472060] Lustre: server umount lustre-MDT0000 complete [ 8628.706264] LustreError: 311227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8628.735651] LustreError: 311227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8633.165670] LustreError: 311208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836426 with bad export cookie 9234788461075996505 [ 8633.170262] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8633.186797] LustreError: 311208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8633.631738] Lustre: server umount lustre-MDT0001 complete [ 8645.041202] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 8648.309746] Key type lgssc unregistered [ 8648.577409] LNet: 313795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8648.587341] LNetError: 313795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8648.614810] LNet: Removed LNI 192.168.206.144@tcp [ 8649.677683] Key type .llcrypt unregistered [ 8649.680079] Key type ._llcrypt unregistered [ 8662.335784] Key type ._llcrypt registered [ 8662.337821] Key type .llcrypt registered [ 8662.440884] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8663.718648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8663.901666] alg: No test for adler32 (adler32-zlib) [ 8665.028358] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 8665.341987] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 8667.023498] Key type lgssc registered [ 8668.343648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8715.791224] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8725.112473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8725.149316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8726.495080] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8726.536234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8726.684217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8731.457470] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8739.733219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8739.808147] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8739.853769] Lustre: 316562:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8740.099263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8744.504195] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8752.193532] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8757.833936] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8777.384758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8777.543249] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8777.766050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8782.992831] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8789.769804] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8792.040974] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8792.042394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8792.046965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8792.058083] Lustre: Skipped 1 previous similar message [ 8792.071525] Lustre: Skipped 1 previous similar message [ 8794.166325] Lustre: server umount lustre-OST0000 complete [ 8801.763844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8801.776565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8801.793052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8803.651641] Lustre: server umount lustre-MDT0000 complete [ 8807.213140] LustreError: 316553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836600 with bad export cookie 16619546947149840580 [ 8807.218545] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8807.224029] LustreError: 316553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8807.563413] Lustre: server umount lustre-MDT0001 complete [ 8817.190667] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 8820.046712] Key type lgssc unregistered [ 8820.308038] LNet: 320136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8820.322210] LNetError: 320136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8820.345090] LNet: Removed LNI 192.168.206.144@tcp [ 8821.402554] Key type .llcrypt unregistered [ 8821.404403] Key type ._llcrypt unregistered [ 8844.293218] Key type ._llcrypt registered [ 8844.295508] Key type .llcrypt registered [ 8844.397590] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 8845.269209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8845.434037] alg: No test for adler32 (adler32-zlib) [ 8846.434702] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 8846.681517] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 8848.344421] Key type lgssc registered [ 8849.322611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8856.334544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8856.350301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8857.909436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8862.453421] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8867.306110] LustreError: 321553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8867.325437] LustreError: 321553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8869.914763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8870.263417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8873.918417] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8880.780109] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8886.568537] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8894.512279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8894.882619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8900.408238] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8907.202195] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8916.448213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8916.467822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8916.497232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8918.496640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8918.500965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8920.348488] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.44@tcp (stopping) [ 8920.352689] Lustre: Skipped 2 previous similar messages [ 8921.610652] Lustre: server umount lustre-MDT0000 complete [ 8923.616820] LustreError: 321554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8923.634164] LustreError: 321554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8925.138553] LustreError: 321533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836718 with bad export cookie 8617539326131053122 [ 8925.144294] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8925.155261] LustreError: 321533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8925.443447] LustreError: 322612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8925.481318] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.44@tcp (stopping) [ 8928.736619] LustreError: 322612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8928.737810] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8928.770701] Lustre: Skipped 1 previous similar message [ 8930.559559] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.44@tcp (stopping) [ 8930.574647] Lustre: Skipped 1 previous similar message [ 8931.590820] Lustre: server umount lustre-MDT0001 complete [ 8937.665792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8945.085574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8950.528454] LustreError: 323384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8950.556439] LustreError: 323384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8950.608918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8950.655715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8954.945927] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8959.856182] LustreError: 325144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8959.867392] LustreError: 325144:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8959.874192] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8962.754735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8963.062994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8967.632933] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 8968.175249] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8974.894756] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8976.482819] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8982.035933] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8983.476862] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8988.647372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8988.659256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8988.668164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8988.696664] Lustre: Skipped 2 previous similar messages [ 8994.828273] Lustre: server umount lustre-OST0000 complete [ 8998.880245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8998.906757] Lustre: Skipped 2 previous similar messages [ 8998.912794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9004.724740] Lustre: server umount lustre-MDT0000 complete [ 9008.535038] LustreError: 321535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836802 with bad export cookie 8617539326131060794 [ 9008.541499] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9008.544585] LustreError: 321535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9008.815656] Lustre: server umount lustre-MDT0001 complete [ 9020.598630] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 9023.913518] Key type lgssc unregistered [ 9024.468143] LNet: 327820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9024.477559] LNetError: 327820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9025.510714] LNet: Removed LNI 192.168.206.144@tcp [ 9026.505347] Key type .llcrypt unregistered [ 9026.510465] Key type ._llcrypt unregistered [ 9053.142245] Key type ._llcrypt registered [ 9053.146413] Key type .llcrypt registered [ 9053.302677] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 9054.621663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9054.820576] alg: No test for adler32 (adler32-zlib) [ 9055.953208] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 9056.171045] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 9057.807171] Key type lgssc registered [ 9058.855524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9066.034639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9066.070518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9067.769852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9072.613371] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9078.242709] LustreError: 329237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9078.252342] LustreError: 329237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9081.633562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9082.128829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9087.422137] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9095.805749] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9101.630315] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9104.721710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9104.729773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9104.743668] Lustre: Skipped 1 previous similar message [ 9104.755918] Lustre: Skipped 1 previous similar message [ 9106.787208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9106.799205] Lustre: Skipped 1 previous similar message [ 9111.125829] Lustre: server umount lustre-MDT0000 complete [ 9112.036604] LustreError: 329237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9112.063281] LustreError: 329237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9115.396333] LustreError: 329217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836909 with bad export cookie 6499225603535360445 [ 9115.408425] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9115.409553] LustreError: 329217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9116.077782] Lustre: server umount lustre-MDT0001 complete [ 9131.441262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9131.887715] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9131.937362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9132.343611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9137.953809] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9146.434738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9146.493619] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9146.514656] Lustre: 331826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9146.700616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9151.116349] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9159.059391] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9164.790716] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9175.538685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9175.855542] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9175.878892] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9175.889672] LustreError: 333708:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9175.895866] LustreError: 333708:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9175.908269] LustreError: 333708:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9176.001906] Lustre: server umount lustre-OST0000 complete [ 9176.005949] LustreError: 333708:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9186.539630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9186.698638] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9187.060145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9192.148291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9194.127935] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9202.654926] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9210.299060] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9210.427068] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9210.519152] Lustre: lustre-OST0001: new disk, initializing [ 9210.529476] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9210.537853] Lustre: 335252:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9210.645974] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9217.170128] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9217.185274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9217.195755] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9217.342374] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9225.039768] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9232.871695] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9232.872924] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9232.886429] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9232.897204] LustreError: Skipped 1 previous similar message [ 9232.922100] Lustre: Skipped 1 previous similar message [ 9234.130904] Lustre: server umount lustre-OST0001 complete [ 9237.999856] LustreError: 334177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9238.021975] LustreError: 334177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9238.496235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9238.513193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9238.558468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9238.569072] Lustre: Skipped 1 previous similar message [ 9243.105054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9243.110525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9243.110657] LustreError: 334178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9243.126101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9243.179834] Lustre: Skipped 1 previous similar message [ 9248.227383] LustreError: 336056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9248.241547] LustreError: 336056:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9252.319355] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9252.659860] Lustre: server umount lustre-OST0000 complete [ 9258.468173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9258.487186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9258.497603] Lustre: Skipped 3 previous similar messages [ 9263.598117] LustreError: 333786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9263.612317] LustreError: 333786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9263.680987] Lustre: server umount lustre-MDT0000 complete [ 9267.689096] LustreError: 331819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837061 with bad export cookie 6499225603535360998 [ 9267.696779] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9267.710347] LustreError: 331819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9267.953512] Lustre: server umount lustre-MDT0001 complete [ 9281.085681] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 9284.063842] Key type lgssc unregistered [ 9284.362571] LNet: 337215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9284.374104] LNetError: 337215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9284.397912] LNet: Removed LNI 192.168.206.144@tcp [ 9285.177166] Key type .llcrypt unregistered [ 9285.181131] Key type ._llcrypt unregistered [ 9352.181403] Key type ._llcrypt registered [ 9352.184102] Key type .llcrypt registered [ 9352.321229] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [ 9369.558675] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 9370.524957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9370.701257] alg: No test for adler32 (adler32-zlib) [ 9371.830755] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 9372.219552] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 9373.999168] Key type lgssc registered [ 9375.410908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9407.280403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9407.306734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9408.542551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9408.644562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9408.760794] Lustre: lustre-MDT0000: new disk, initializing [ 9408.855278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9408.878616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9413.795227] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9423.403116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9423.513962] Lustre: 342148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9423.561880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9423.565448] Lustre: Skipped 1 previous similar message [ 9423.681161] Lustre: lustre-MDT0001: new disk, initializing [ 9423.786565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9423.825042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9423.846597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9428.466042] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9436.271740] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9441.451204] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9448.561444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9448.956289] Lustre: lustre-OST0000: new disk, initializing [ 9448.961205] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9448.968215] Lustre: 344117:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9449.099605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9456.667753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9456.676868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9456.763063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9456.905875] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9465.379105] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9469.709641] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9469.956613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9473.836959] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9474.036596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9477.093632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9477.093708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9477.100543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9477.124477] Lustre: Skipped 1 previous similar message [ 9481.811845] Lustre: server umount lustre-OST0000 complete [ 9485.282105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9485.287149] LustreError: Skipped 1 previous similar message [ 9485.292228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9485.308962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9485.313126] Lustre: Skipped 1 previous similar message [ 9487.327938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9487.333185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9491.225658] Lustre: server umount lustre-MDT0000 complete [ 9495.531240] LustreError: 342953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837289 with bad export cookie 4258244523415245957 [ 9495.539020] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9495.924033] Lustre: server umount lustre-MDT0001 complete [ 9503.097924] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 09:28:15 (1787837295) [ 9509.323194] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 09:28:21 (1787837301) [ 9517.258696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9517.708178] LustreError: 346872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9517.744745] LustreError: 346872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9517.801161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9522.423710] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9523.171235] LustreError: 346873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9527.269798] LustreError: 346872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9531.038633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9531.500492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9536.042469] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9542.527857] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9548.431647] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9555.081585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9555.396059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9561.505498] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9568.090130] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9575.542497] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9576.974092] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9578.512381] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9580.085730] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9581.700172] Lustre: Failing over lustre-MDT0000 [ 9581.946328] Lustre: server umount lustre-MDT0000 complete [ 9582.562309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9582.570517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9582.591839] LustreError: 346872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9588.192858] LustreError: 346872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9588.214588] LustreError: 346872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9598.434457] Lustre: 340373:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787837376/real 1787837376] req@00000000ae453202 x1874683143194368/t0(0) o400->MGC192.168.206.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787837392 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9598.444687] LustreError: 346867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9598.486787] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9598.525421] LustreError: 346867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9600.824091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9608.687739] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3b1852d538eb1c2d [ 9608.694182] Lustre: MGC192.168.206.144@tcp: Connection restored to 0@lo (at 0@lo) [ 9608.935667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9608.999803] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9611.526456] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9613.195150] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9614.343921] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9614.408355] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9614.465359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9622.757150] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9624.185440] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9627.647588] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9629.311548] Lustre: Failing over lustre-MDT0000 [ 9629.455987] LustreError: 350835:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (000000007e910e74) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 9629.664780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9629.674763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9629.678887] LustreError: 350700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9629.678897] LustreError: 350700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 9629.730644] Lustre: Skipped 4 previous similar messages [ 9631.560114] Lustre: server umount lustre-MDT0000 complete [ 9649.050255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9649.136364] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9649.326760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9649.379098] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9653.392872] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9653.502436] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9654.766950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9654.782195] Lustre: Skipped 2 previous similar messages [ 9654.822614] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9654.903240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9661.944975] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9663.404735] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9670.122323] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9670.123843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9670.124740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9670.124745] Lustre: Skipped 1 previous similar message [ 9670.137216] Lustre: Skipped 2 previous similar messages [ 9675.175939] Lustre: server umount lustre-MDT0000 complete [ 9675.236269] LustreError: 346867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9675.247078] LustreError: 346867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 9678.779191] LustreError: 346852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837472 with bad export cookie 4258244523415248673 [ 9678.799861] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9678.800705] LustreError: 346852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9679.307526] Lustre: server umount lustre-MDT0001 complete [ 9692.625444] Lustre: server umount lustre-OST0000 complete [ 9707.172859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9711.693583] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9719.349616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9719.676987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9719.683821] Lustre: Skipped 1 previous similar message [ 9723.298491] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9729.557344] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9734.546538] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9740.574679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9746.582698] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9753.247452] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9754.231209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9764.834379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9764.844617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9764.850280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9764.856974] Lustre: Skipped 1 previous similar message [ 9764.863711] Lustre: Skipped 2 previous similar messages [ 9768.299368] Lustre: server umount lustre-OST0000 complete [ 9775.073663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9775.086632] Lustre: Skipped 2 previous similar messages [ 9777.952276] Lustre: server umount lustre-MDT0000 complete [ 9780.194341] LustreError: 353207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9780.210885] LustreError: 353207:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9781.269966] LustreError: 353190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837575 with bad export cookie 4258244523415249394 [ 9781.276843] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9781.285218] LustreError: 353190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9781.632519] Lustre: server umount lustre-MDT0001 complete [ 9791.530378] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [ 9794.364831] Key type lgssc unregistered [ 9794.622789] LNet: 356747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9794.633531] LNetError: 356747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9795.688918] LNet: Removed LNI 192.168.206.144@tcp [ 9796.464406] Key type .llcrypt unregistered [ 9796.466864] Key type ._llcrypt unregistered [ 9818.914214] Key type ._llcrypt registered [ 9818.916237] Key type .llcrypt registered [ 9819.050207] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [ 9820.059187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9820.120294] alg: No test for adler32 (adler32-zlib) [ 9821.297887] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 9821.576697] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [ 9823.255189] Key type lgssc registered [ 9824.411270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9830.958401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9830.974795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9832.497403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9837.520382] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9842.656930] LustreError: 358244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9842.677515] LustreError: 358244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9845.400770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9845.755253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9850.346362] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9857.296985] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9862.181990] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9867.740127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9868.098092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9874.207290] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9877.304217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9880.873603] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9891.810907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9891.823645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9891.836024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9892.840603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9892.845404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9893.431391] Lustre: server umount lustre-MDT0000 complete [ 9897.186739] LustreError: 358225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837690 with bad export cookie 7623733205550654868 [ 9897.202639] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9897.203087] LustreError: 358225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9897.638499] Lustre: server umount lustre-MDT0001 complete [ 9911.952042] Lustre: server umount lustre-OST0000 complete [ 9924.319800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9924.890686] LustreError: 361955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9924.991060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9929.651358] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9930.209983] LustreError: 361956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9934.307576] LustreError: 361955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9939.032858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9939.606981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9945.269288] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9952.858372] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9958.499898] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9964.923483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9965.226854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9969.337834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9971.908285] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [ 9978.879362] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9995.240146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9995.241715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9995.257337] Lustre: Skipped 2 previous similar messages [ 9995.269926] Lustre: Skipped 2 previous similar messages [ 9997.076322] Lustre: server umount lustre-MDT0000 complete [10000.231082] LustreError: 361937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837793 with bad export cookie 7623733205550655981 [10000.232899] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10000.244744] LustreError: 361937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10000.352180] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10000.352222] LustreError: 361950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10000.359059] Lustre: Skipped 1 previous similar message [10000.361652] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10000.724969] Lustre: server umount lustre-MDT0001 complete [10007.016478] Lustre: server umount lustre-OST0000 complete [10021.214622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10021.927986] LustreError: 365865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10022.034747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10027.014366] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10034.807202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10038.963042] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10045.162187] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10050.722934] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10057.132734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10057.466715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10057.472666] Lustre: Skipped 1 previous similar message [10063.072719] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10066.682060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10069.809504] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10075.107052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10075.118038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10075.140895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10077.156083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10080.219778] Lustre: server umount lustre-OST0000 complete [10086.368102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10086.381156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10086.399421] Lustre: Skipped 1 previous similar message [10086.404763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10086.409595] Lustre: Skipped 2 previous similar messages [10090.181087] Lustre: server umount lustre-MDT0000 complete [10092.512675] LustreError: 367106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10092.529487] LustreError: 367106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10093.291693] LustreError: 367709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837887 with bad export cookie 7623733205550657003 [10093.294450] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10093.304519] LustreError: 367709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10093.554660] Lustre: server umount lustre-MDT0001 complete [10103.318735] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [10105.716858] Key type lgssc unregistered [10105.944439] LNet: 369399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10105.961545] LNetError: 369399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10105.980914] LNet: Removed LNI 192.168.206.144@tcp [10106.766787] Key type .llcrypt unregistered [10106.768824] Key type ._llcrypt unregistered [10127.940632] Key type ._llcrypt registered [10127.942981] Key type .llcrypt registered [10128.027799] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [10128.779992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10128.805040] alg: No test for adler32 (adler32-zlib) [10129.870017] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [10130.141765] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [10131.823214] Key type lgssc registered [10132.906391] Lustre: Echo OBD driver; http://www.lustre.org/ [10139.465952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10139.484049] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10139.493756] LustreError: 370819:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10139.507323] LustreError: 370819:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10139.521351] LustreError: 370819:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10139.532652] LustreError: 370819:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10139.551432] LustreError: 370819:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10146.418638] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10146.422129] LustreError: 371217:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10146.434178] LustreError: 371217:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10146.441135] LustreError: 371217:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10146.449594] LustreError: 371217:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10146.457140] LustreError: 371217:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10164.201373] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [10166.648639] Key type lgssc unregistered [10166.895203] LNet: 372222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10166.905883] LNetError: 372222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10166.933901] LNet: Removed LNI 192.168.206.144@tcp [10167.662517] Key type .llcrypt unregistered [10167.666092] Key type ._llcrypt unregistered [10186.167482] Key type ._llcrypt registered [10186.170417] Key type .llcrypt registered [10186.279527] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [10199.644364] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [10200.372402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10200.417457] alg: No test for adler32 (adler32-zlib) [10201.562837] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [10201.822991] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [10203.568145] Key type lgssc registered [10204.875344] Lustre: Echo OBD driver; http://www.lustre.org/ [10241.236455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10241.276655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10242.570118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10242.589688] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10242.668577] Lustre: lustre-MDT0000: new disk, initializing [10242.799239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10242.814144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10247.199766] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10256.544692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10256.732132] Lustre: 375380:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10256.812825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10256.821384] Lustre: Skipped 1 previous similar message [10256.918508] Lustre: lustre-MDT0001: new disk, initializing [10257.025585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10257.054242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10257.072648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10261.064313] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10271.309641] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10278.595138] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10285.769560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10286.122361] Lustre: lustre-OST0000: new disk, initializing [10286.137793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10286.160680] Lustre: 377346:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10286.257419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10291.440650] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10292.250579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10292.256596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10292.369797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10299.332878] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10302.795561] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10302.987217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10307.270985] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10307.460068] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10312.672049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10312.678905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10312.695793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10312.701741] Lustre: Skipped 1 previous similar message [10315.152377] Lustre: server umount lustre-OST0000 complete [10318.304251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10318.310715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10318.323608] Lustre: Skipped 1 previous similar message [10318.327178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10322.915241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10322.916248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10324.485953] Lustre: server umount lustre-MDT0000 complete [10328.151169] LustreError: 377345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838121 with bad export cookie 6429064955020519346 [10328.168576] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10328.170774] LustreError: 377345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10328.438186] Lustre: server umount lustre-MDT0001 complete [10335.859202] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10337.365904] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 09:42:09 (1787838129) [10343.573352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10344.168127] LustreError: 379806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10344.198207] LustreError: 379806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10344.249313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10348.174905] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10349.548184] LustreError: 379807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10353.633215] LustreError: 379806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10355.588565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10355.883848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10360.131799] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10366.723342] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10371.763939] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10377.749532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10378.023753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10384.274838] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10391.668408] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10399.024898] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10399.336181] Lustre: lustre-OST0001: new disk, initializing [10399.347552] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10399.355765] Lustre: 382644:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10399.518870] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10404.970561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10404.984325] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10405.163271] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10406.838486] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10416.394845] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10420.710143] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10420.714223] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10420.720245] LustreError: Skipped 1 previous similar message [10420.728544] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10420.734856] Lustre: Skipped 1 previous similar message [10420.768890] Lustre: Skipped 2 previous similar messages [10425.856929] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10425.867129] Lustre: Skipped 2 previous similar messages [10426.033964] Lustre: server umount lustre-OST0001 complete [10428.173078] LustreError: 382346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10436.069891] LustreError: 381630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10436.105850] LustreError: 381630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10444.770586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10444.791276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10444.804049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10446.319536] LustreError: 382346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10446.340569] LustreError: 382346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10449.833447] Lustre: server umount lustre-OST0000 complete [10453.473253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10453.482863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10453.495950] Lustre: Skipped 1 previous similar message [10459.794700] Lustre: server umount lustre-MDT0000 complete [10463.654236] LustreError: 379788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838257 with bad export cookie 6429064955020520228 [10463.656048] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10463.665023] LustreError: 379788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10470.079304] Lustre: server umount lustre-MDT0001 complete [10479.593111] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [10482.483888] Key type lgssc unregistered [10482.779604] LNet: 384631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10482.794315] LNetError: 384631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10482.818091] LNet: Removed LNI 192.168.206.144@tcp [10483.821891] Key type .llcrypt unregistered [10483.823958] Key type ._llcrypt unregistered [10536.132506] Key type ._llcrypt registered [10536.135770] Key type .llcrypt registered [10536.283171] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [10537.681488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10537.705081] alg: No test for adler32 (adler32-zlib) [10538.758628] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [10539.030759] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [10540.767168] Key type lgssc registered [10542.082557] Lustre: Echo OBD driver; http://www.lustre.org/ [10550.355219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10550.370063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10551.752870] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10551.789750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10552.050726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10555.821104] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10562.997886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10563.062276] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10563.083787] Lustre: 387431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10563.239133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10566.844776] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10573.966953] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10579.720237] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10582.498222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10582.507800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10587.617037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10587.628746] Lustre: Skipped 2 previous similar messages [10588.051583] Lustre: server umount lustre-MDT0000 complete [10591.712205] LustreError: 388183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838385 with bad export cookie 3914644524957950602 [10591.720753] LustreError: 388183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10591.724012] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10591.986871] Lustre: server umount lustre-MDT0001 complete [10599.979923] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 09:46:32 (1787838392) [10606.445059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10607.001726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10610.786437] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10612.192770] LustreError: 389944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10612.223645] LustreError: 389944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10617.320211] LustreError: 389943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10618.287674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10618.632733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10623.345268] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10630.975877] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10636.201399] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10643.147360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10643.349626] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10643.589957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10650.115392] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10658.661864] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10665.920303] Lustre: Permanently deactivating lustre-OST0000 [10665.934125] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10673.689480] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10675.926571] LustreError: 392976:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10675.937777] LustreError: 392976:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10675.951751] LustreError: 392976:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10678.206604] LustreError: 393025:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10678.216842] LustreError: 393025:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10685.857414] Lustre: server umount lustre-OST0000 complete [10688.358739] LustreError: 393275:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10688.363389] LustreError: 393275:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10690.528711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10690.536957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10690.546643] Lustre: Skipped 1 previous similar message [10690.558433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10693.601766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10693.603292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10693.627204] Lustre: Skipped 1 previous similar message [10698.722331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10698.728458] Lustre: Skipped 1 previous similar message [10704.351617] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10704.494567] Lustre: server umount lustre-MDT0000 complete [10707.586101] LustreError: 389923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838501 with bad export cookie 3914644524957951183 [10707.598216] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10707.608177] LustreError: 389923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10718.379570] Lustre: server umount lustre-MDT0001 complete [10725.628419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10729.986813] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10732.494638] LustreError: 394435:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10738.400039] Lustre: 394774:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.144@tcp [10740.392587] Lustre: server umount MGS complete [10751.898545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10752.306211] LustreError: 395498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10752.318702] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10752.330051] Lustre: Skipped 1 previous similar message [10752.385841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10756.424718] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10757.601507] LustreError: 395499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10762.727096] LustreError: 395498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10764.674279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10765.044341] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10769.460988] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10776.114878] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10781.082273] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10782.920256] Lustre: Permanently reactivating lustre-OST0000 [10782.937425] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10782.944976] Lustre: Skipped 2 previous similar messages [10809.311734] LustreError: 397247:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10817.829692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10818.275529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10818.286929] Lustre: Skipped 1 previous similar message [10824.985539] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10831.855141] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10839.018798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10839.034107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10839.051601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10839.063698] Lustre: Skipped 2 previous similar messages [10839.238100] Lustre: server umount lustre-OST0000 complete [10845.160632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10845.173132] Lustre: Skipped 1 previous similar message [10852.423344] Lustre: server umount lustre-MDT0000 complete [10854.371697] LustreError: 398822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10854.389405] LustreError: 398822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10856.066087] LustreError: 395479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838649 with bad export cookie 3914644524957952233 [10856.075436] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10856.082413] LustreError: 395479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10856.440176] Lustre: server umount lustre-MDT0001 complete [10867.202760] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [10870.241878] Key type lgssc unregistered [10870.629183] LNet: 399863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10870.643223] LNetError: 399863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10870.661176] LNet: Removed LNI 192.168.206.144@tcp [10871.559972] Key type .llcrypt unregistered [10871.564059] Key type ._llcrypt unregistered [10890.053418] Key type ._llcrypt registered [10890.056312] Key type .llcrypt registered [10890.166907] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [10905.461519] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [10907.463261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10907.531816] alg: No test for adler32 (adler32-zlib) [10908.926675] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [10909.448567] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [10911.271887] Key type lgssc registered [10912.809153] Lustre: Echo OBD driver; http://www.lustre.org/ [10952.769480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10952.811910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10954.088123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10954.112022] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10954.174828] Lustre: lustre-MDT0000: new disk, initializing [10954.239513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10954.253841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10959.170854] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10969.205450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10969.328720] Lustre: 403020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10969.402701] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10969.406806] Lustre: Skipped 1 previous similar message [10969.475720] Lustre: lustre-MDT0001: new disk, initializing [10969.566353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10969.606044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10969.622506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10974.097553] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [10983.987211] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10989.858339] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10996.301960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10996.594955] Lustre: lustre-OST0000: new disk, initializing [10996.600778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10996.611770] Lustre: 404986:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10996.730030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11003.232589] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11003.500238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11003.525953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11003.632876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11012.124202] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11016.430553] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11016.599932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11021.731911] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11022.015367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11024.354966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11024.359231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11024.360235] LustreError: Skipped 1 previous similar message [11024.375049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11024.383680] Lustre: Skipped 1 previous similar message [11024.400887] Lustre: Skipped 1 previous similar message [11029.478572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11029.483842] Lustre: Skipped 1 previous similar message [11034.592732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11038.175129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11038.487246] Lustre: server umount lustre-OST0000 complete [11044.834643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11044.846096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11044.852152] Lustre: Skipped 2 previous similar messages [11048.672925] Lustre: server umount lustre-MDT0000 complete [11049.956047] LustreError: 403025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11049.968936] LustreError: 403025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11053.082917] LustreError: 403010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838846 with bad export cookie 15370291845137072969 [11053.086465] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11053.091840] LustreError: 403010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11053.439918] Lustre: server umount lustre-MDT0001 complete [11060.848914] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11062.243177] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 09:54:14 (1787838854) [11068.069210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11068.542836] LustreError: 407450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11068.633291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11073.352908] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11074.019681] LustreError: 407451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11079.138862] LustreError: 407450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11081.271441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11081.552919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11086.107679] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11093.879073] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11099.723697] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11106.686292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11107.021107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11113.306126] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11120.057478] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11121.849690] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11124.444050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11133.926106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11133.935161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11133.955068] Lustre: Skipped 1 previous similar message [11133.964861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11134.956879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11136.774576] Lustre: server umount lustre-OST0000 complete [11143.138437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11143.148956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11143.158932] Lustre: Skipped 1 previous similar message [11143.163678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11143.168858] Lustre: Skipped 2 previous similar messages [11146.831228] Lustre: server umount lustre-MDT0000 complete [11150.306917] LustreError: 408178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11150.319867] LustreError: 408178:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11150.913231] LustreError: 407432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838944 with bad export cookie 15370291845137073851 [11150.919690] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11150.931316] LustreError: 407432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11151.309313] Lustre: server umount lustre-MDT0001 complete [11163.198992] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [11166.127203] Key type lgssc unregistered [11166.414419] LNet: 411178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11166.426769] LNetError: 411178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11166.447603] LNet: Removed LNI 192.168.206.144@tcp [11167.321931] Key type .llcrypt unregistered [11167.324936] Key type ._llcrypt unregistered [11193.451508] Key type ._llcrypt registered [11193.455320] Key type .llcrypt registered [11193.574914] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [11194.172514] Key type .llcrypt unregistered [11194.180415] Key type ._llcrypt unregistered [11212.889399] Key type ._llcrypt registered [11212.890959] Key type .llcrypt registered [11212.986707] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [11214.345678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11214.381264] alg: No test for adler32 (adler32-zlib) [11215.443902] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [11215.722659] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [11217.383152] Key type lgssc registered [11218.778382] Lustre: Echo OBD driver; http://www.lustre.org/ [11226.340888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11226.362240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11227.901681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11232.314789] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11237.354597] LustreError: 413394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11237.367415] LustreError: 413394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11240.658329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11241.065269] LustreError: 413395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11241.095220] LustreError: 413395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11241.108095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11246.570966] LustreError: 413390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11246.592154] LustreError: 413390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11247.489267] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11251.682106] LustreError: 413391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11251.700966] LustreError: 413391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11252.737877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11254.618943] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11257.825451] LustreError: 413394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11257.841751] LustreError: 413394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11261.035893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11261.482456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11266.461111] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11276.481622] Lustre: server umount lustre-OST0000 complete [11281.889484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11281.889779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11281.895700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11281.912552] Lustre: Skipped 1 previous similar message [11281.922640] Lustre: Skipped 1 previous similar message [11287.011101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11287.015359] Lustre: Skipped 1 previous similar message [11290.174130] Lustre: server umount lustre-MDT0000 complete [11292.129504] LustreError: 414139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11292.156547] LustreError: 414139:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11293.154443] Lustre: 412708:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787839070/real 1787839070] req@00000000ba9c9d87 x1874685076727168/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787839086 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11293.194563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11293.922179] LustreError: 413376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839087 with bad export cookie 16041949767943814307 [11293.923899] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11293.929923] LustreError: 413376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11294.208545] Lustre: server umount lustre-MDT0001 complete [11303.728028] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [11306.209025] Key type lgssc unregistered [11306.467931] LNet: 416543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11306.480295] LNetError: 416543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11306.505545] LNet: Removed LNI 192.168.206.144@tcp [11307.288232] Key type .llcrypt unregistered [11307.289725] Key type ._llcrypt unregistered [11329.953539] Key type ._llcrypt registered [11329.955801] Key type .llcrypt registered [11330.076666] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [11331.568795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11331.747096] alg: No test for adler32 (adler32-zlib) [11332.824282] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [11333.135028] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [11334.815155] Key type lgssc registered [11335.889755] Lustre: Echo OBD driver; http://www.lustre.org/ [11342.583222] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11342.608748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11358.175388] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11363.297946] LustreError: 417934:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.144@tcp: failed processing log, type 4: rc = -110 [11389.983213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11395.277841] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11401.364987] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11407.161722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11407.630851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11407.694547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11410.948869] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11412.962651] LustreError: 417959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11412.992282] LustreError: 417959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11417.700889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11418.082337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11421.784211] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11428.588873] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11433.875620] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11443.285335] Lustre: server umount lustre-OST0000 complete [11448.802914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11448.812032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11448.846565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11450.849640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11450.853785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11450.872874] Lustre: Skipped 1 previous similar message [11455.970386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11455.979650] Lustre: Skipped 1 previous similar message [11458.091707] Lustre: server umount lustre-MDT0000 complete [11459.999095] Lustre: 417271:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787839237/real 1787839237] req@0000000004369b27 x1874685199414144/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787839253 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11460.034304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11460.060048] LustreError: 418976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11460.076029] LustreError: 418976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11461.379738] LustreError: 417940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839255 with bad export cookie 13127860477629807408 [11461.380979] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11461.392849] LustreError: 417940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11461.643202] Lustre: server umount lustre-MDT0001 complete [11470.216849] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [11472.437189] Key type lgssc unregistered [11472.697774] LNet: 421518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11472.708650] LNetError: 421518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11472.719918] LNet: Removed LNI 192.168.206.144@tcp [11473.476174] Key type .llcrypt unregistered [11473.478884] Key type ._llcrypt unregistered [11492.266418] Key type ._llcrypt registered [11492.270746] Key type .llcrypt registered [11492.366155] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [11493.445707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11493.553453] alg: No test for adler32 (adler32-zlib) [11494.589827] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [11494.826779] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [11496.488051] Key type lgssc registered [11497.423223] Lustre: Echo OBD driver; http://www.lustre.org/ [11503.530452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11503.549909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11504.848658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11508.300643] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11514.340383] LustreError: 422912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11514.358541] LustreError: 422912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11515.825200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11516.192444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11520.027698] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11528.440959] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11533.839808] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11540.622535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11541.087889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11547.370857] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11550.206185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11555.230674] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11560.417833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11560.434111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11560.440382] Lustre: Skipped 2 previous similar messages [11560.456870] Lustre: Skipped 2 previous similar messages [11562.271675] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.44@tcp (stopping) [11564.816799] Lustre: server umount lustre-MDT0000 complete [11565.543530] LustreError: 422909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11565.569330] LustreError: 422909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11567.269950] LustreError: 423647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.44@tcp arrived at 1787839361 with bad export cookie 16906224373458623533 [11567.285546] LustreError: 423647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11569.066621] LustreError: 422894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839362 with bad export cookie 16906224373458622637 [11569.074101] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11569.080206] LustreError: 422894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11570.657567] LustreError: 423654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11570.659781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11570.673281] LustreError: 423654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11570.693326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11574.692022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11574.693266] LustreError: 422908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11574.734239] LustreError: 422908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11575.392590] Lustre: server umount lustre-OST0000 complete [11578.786309] LustreError: 422909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11578.803223] LustreError: 422909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11582.785967] Lustre: server umount lustre-MDT0001 complete [11593.192825] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [11596.315839] Key type lgssc unregistered [11596.575191] LNet: 426603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11596.584835] LNetError: 426603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11596.598917] LNet: Removed LNI 192.168.206.144@tcp [11597.646270] Key type .llcrypt unregistered [11597.650271] Key type ._llcrypt unregistered [11619.686716] Key type ._llcrypt registered [11619.689491] Key type .llcrypt registered [11619.770630] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [11620.700759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11620.876901] alg: No test for adler32 (adler32-zlib) [11621.914596] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [11622.151627] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [11623.847378] Key type lgssc registered [11625.008474] Lustre: Echo OBD driver; http://www.lustre.org/ [11631.867701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11631.886839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11633.255081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11636.959722] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11642.851921] LustreError: 428019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11642.869836] LustreError: 428019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11644.004265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11644.398572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11648.890540] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11655.339424] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11659.886358] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11665.366648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11665.631465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11670.634655] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11677.949127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11679.671094] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11685.344454] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11685.357527] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11685.381347] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11686.680858] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.44@tcp (stopping) [11688.422646] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11688.442457] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11690.025836] Lustre: server umount lustre-MDT0001 complete [11691.791447] LustreError: 429260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11693.548741] LustreError: 428016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11693.576179] LustreError: 428016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11698.658115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11698.658439] LustreError: 428020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11698.669716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11698.700394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11698.704152] Lustre: Skipped 1 previous similar message [11703.615977] Lustre: server umount lustre-OST0000 complete [11703.783508] LustreError: 428747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11707.224350] Lustre: server umount lustre-MDT0000 complete [11719.445565] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [11721.959357] Key type lgssc unregistered [11722.253663] LNet: 431705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11722.257279] LNetError: 431705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11722.281786] LNet: Removed LNI 192.168.206.144@tcp [11723.334821] Key type .llcrypt unregistered [11723.338982] Key type ._llcrypt unregistered [11744.284648] Key type ._llcrypt registered [11744.287493] Key type .llcrypt registered [11744.450245] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [11759.002555] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [11760.453667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11760.522106] alg: No test for adler32 (adler32-zlib) [11761.530968] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [11761.725068] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [11763.439205] Key type lgssc registered [11764.590642] Lustre: Echo OBD driver; http://www.lustre.org/ [11803.294345] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [11819.988994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11820.007727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11821.270697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11821.317714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11821.376331] Lustre: lustre-MDT0000: new disk, initializing [11821.428390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11821.445087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11826.058173] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11839.375412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11839.708263] Lustre: lustre-OST0000: new disk, initializing [11839.712257] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [11839.718358] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11839.728323] Lustre: Skipped 1 previous similar message [11839.731707] Lustre: 436741:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11839.823242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11845.150692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11845.166833] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11845.207781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11845.463676] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11853.237904] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11867.239720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11867.364117] Lustre: 435721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11867.459078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11867.533356] Lustre: lustre-MDT0001: new disk, initializing [11867.616515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11867.655332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11867.664202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11871.253588] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [11880.294252] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11880.510612] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11885.297500] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11885.483673] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11891.176642] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11891.179136] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11891.179632] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11891.201639] Lustre: Skipped 1 previous similar message [11893.536592] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.44@tcp (stopping) [11893.557987] Lustre: Skipped 1 previous similar message [11896.292804] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11896.312808] Lustre: Skipped 1 previous similar message [11897.137867] Lustre: server umount lustre-MDT0001 complete [11906.536535] LustreError: 435734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11906.570368] LustreError: 435734:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11911.657727] LustreError: 435729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11911.685457] LustreError: 435729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11916.770742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11916.775396] LustreError: 438398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11916.779678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11916.781759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11921.896347] LustreError: 435733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11927.015646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11927.017069] LustreError: 438398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11927.027915] Lustre: Skipped 1 previous similar message [11929.055122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11929.175200] Lustre: server umount lustre-OST0000 complete [11939.072252] Lustre: server umount lustre-MDT0000 complete [11949.638320] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [11951.955568] Key type lgssc unregistered [11952.222973] LNet: 440491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11952.230493] LNetError: 440491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11952.247183] LNet: Removed LNI 192.168.206.144@tcp [11952.829882] Key type .llcrypt unregistered [11952.831628] Key type ._llcrypt unregistered [11983.780834] Key type ._llcrypt registered [11983.783754] Key type .llcrypt registered [11983.899019] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [11984.231964] Key type .llcrypt unregistered [11984.237453] Key type ._llcrypt unregistered [11998.900098] Key type ._llcrypt registered [11998.902839] Key type .llcrypt registered [11999.050427] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [12000.263805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12000.303083] alg: No test for adler32 (adler32-zlib) [12001.403220] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [12001.638682] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [12003.287124] Key type lgssc registered [12004.325110] Lustre: Echo OBD driver; http://www.lustre.org/ [12027.731951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12027.779433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12028.977265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12029.014234] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12029.109787] Lustre: lustre-MDT0000: new disk, initializing [12029.224463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12029.242390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12033.129154] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12043.572127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12043.773191] Lustre: 443862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12043.891760] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12043.902079] Lustre: Skipped 1 previous similar message [12044.055552] Lustre: lustre-MDT0001: new disk, initializing [12044.178536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12044.236993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12044.255621] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12049.061431] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12059.484159] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12064.435759] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12070.605672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12070.928272] Lustre: lustre-OST0000: new disk, initializing [12070.930797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12070.940709] Lustre: 445829:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12071.040584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12076.317404] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12078.133231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12078.143989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12078.262888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12085.646316] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12093.923777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12093.933697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12093.936374] Lustre: Skipped 2 previous similar messages [12093.956038] Lustre: Skipped 2 previous similar messages [12097.723465] Lustre: server umount lustre-MDT0000 complete [12101.333828] LustreError: 443856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839895 with bad export cookie 17479972678782640013 [12101.337213] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12101.349756] LustreError: 443856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12101.740131] Lustre: server umount lustre-MDT0001 complete [12115.927828] Lustre: server umount lustre-OST0000 complete [12124.850837] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 10:11:57 (1787839917) [12131.068673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12131.509352] LustreError: 447861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12131.526388] LustreError: 447861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12131.571736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12135.066663] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12136.929700] LustreError: 447862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12141.479835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12141.829415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12146.021433] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12151.760819] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12156.648763] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12176.049221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12176.221144] Lustre: Found index 0 for lustre-OST0000, updating log [12176.398262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12177.454860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12181.338600] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12186.466848] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12192.739750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12192.739782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12192.740779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12192.748359] LustreError: Skipped 1 previous similar message [12192.770590] Lustre: Skipped 1 previous similar message [12193.826831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12193.832488] Lustre: Skipped 2 previous similar messages [12195.565699] Lustre: server umount lustre-OST0000 complete [12209.913531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12209.914074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12209.921564] Lustre: Skipped 1 previous similar message [12209.928169] Lustre: Skipped 1 previous similar message [12215.264622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12215.266656] Lustre: Skipped 1 previous similar message [12216.145107] Lustre: server umount lustre-MDT0000 complete [12218.761948] LustreError: 449704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840012 with bad export cookie 17479972678782641406 [12218.767135] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12218.771386] LustreError: 449704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12219.049788] Lustre: server umount lustre-MDT0001 complete [12225.678856] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 10:13:37 (1787840017) [12239.200090] LNet: 1 local NIs in recovery (showing 1): 192.168.206.144@tcp [12840.835670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12841.229689] LustreError: 454269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12841.321411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12845.010382] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12846.565103] LustreError: 454270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12851.686495] LustreError: 454269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12851.874255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12852.228268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12855.637219] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12861.860935] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12867.448615] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12886.829517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12887.067815] Lustre: Found index 0 for lustre-OST0000, updating log [12887.326846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12890.414782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [12891.952822] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [12892.511480] LNet: 1 local NIs in recovery (showing 1): 192.168.206.144@tcp [12897.245481] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12899.813914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12899.828755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12913.119117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12913.271657] Lustre: server umount lustre-OST0000 complete [12917.217705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12917.221084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12917.234718] Lustre: Skipped 1 previous similar message [12917.258408] Lustre: Skipped 1 previous similar message [12919.335699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12919.347200] Lustre: Skipped 3 previous similar messages [12922.407507] LustreError: 454269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12922.535445] Lustre: server umount lustre-MDT0000 complete [12925.729863] LustreError: 454248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840719 with bad export cookie 17479972678782642484 [12925.740607] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12926.056852] Lustre: server umount lustre-MDT0001 complete [12957.024198] LNet: 1 local NIs in recovery (showing 1): 192.168.206.144@tcp [12960.091305] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [12970.039618] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [13002.371720] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 10:26:34 (1787840794) [13016.279177] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [13018.963924] Key type lgssc unregistered [13019.326544] LNet: 460879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13019.333705] LNetError: 460879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13149.415577] LNet: Removed LNI 192.168.206.144@tcp [13150.086735] Key type .llcrypt unregistered [13150.090974] Key type ._llcrypt unregistered [13163.416980] Key type ._llcrypt registered [13163.418494] Key type .llcrypt registered [13163.547355] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [13164.250180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13164.375064] alg: No test for adler32 (adler32-zlib) [13165.355883] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [13165.527923] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [13167.215714] Key type lgssc registered [13168.509154] Lustre: Echo OBD driver; http://www.lustre.org/ [13172.397857] LNet: Added LNI 192.168.206.144@tcp100 [8/256/0/180] [13173.388476] LNet: Added LNI 192.168.206.144@tcp101 [8/256/0/180] [13174.376866] LNet: Added LNI 192.168.206.144@tcp102 [8/256/0/180] [13175.205791] LNet: Added LNI 192.168.206.144@tcp103 [8/256/0/180] [13176.441056] LNet: Added LNI 192.168.206.144@tcp104 [8/256/0/180] [13177.436547] LNet: Added LNI 192.168.206.144@tcp105 [8/256/0/180] [13178.471123] LNet: Added LNI 192.168.206.144@tcp106 [8/256/0/180] [13179.485573] LNet: Added LNI 192.168.206.144@tcp107 [8/256/0/180] [13180.364265] LNet: Added LNI 192.168.206.144@tcp108 [8/256/0/180] [13181.208821] LNet: Added LNI 192.168.206.144@tcp109 [8/256/0/180] [13182.405981] LNet: Added LNI 192.168.206.144@tcp110 [8/256/0/180] [13183.695988] LNet: Added LNI 192.168.206.144@tcp111 [8/256/0/180] [13184.981327] LNet: Added LNI 192.168.206.144@tcp112 [8/256/0/180] [13186.166552] LNet: Added LNI 192.168.206.144@tcp113 [8/256/0/180] [13187.645138] LNet: Added LNI 192.168.206.144@tcp114 [8/256/0/180] [13188.855563] LNet: Added LNI 192.168.206.144@tcp115 [8/256/0/180] [13190.336403] LNet: Added LNI 192.168.206.144@tcp116 [8/256/0/180] [13191.476532] LNet: Added LNI 192.168.206.144@tcp117 [8/256/0/180] [13193.013189] LNet: Added LNI 192.168.206.144@tcp118 [8/256/0/180] [13193.927721] LNet: Added LNI 192.168.206.144@tcp119 [8/256/0/180] [13194.561650] LNet: Added LNI 192.168.206.144@tcp120 [8/256/0/180] [13195.695645] LNet: Added LNI 192.168.206.144@tcp121 [8/256/0/180] [13196.783507] LNet: Added LNI 192.168.206.144@tcp122 [8/256/0/180] [13197.563629] LNet: Added LNI 192.168.206.144@tcp123 [8/256/0/180] [13198.456846] LNet: Added LNI 192.168.206.144@tcp124 [8/256/0/180] [13199.318507] LNet: Added LNI 192.168.206.144@tcp125 [8/256/0/180] [13200.302872] LNet: Added LNI 192.168.206.144@tcp126 [8/256/0/180] [13201.249290] LNet: Added LNI 192.168.206.144@tcp127 [8/256/0/180] [13202.047371] LNet: Added LNI 192.168.206.144@tcp128 [8/256/0/180] [13202.749049] LNet: Added LNI 192.168.206.144@tcp129 [8/256/0/180] [13203.849865] LNet: Added LNI 192.168.206.144@tcp130 [8/256/0/180] [13204.947025] LNet: Added LNI 192.168.206.144@tcp131 [8/256/0/180] [13206.117977] LNet: Added LNI 192.168.206.144@tcp132 [8/256/0/180] [13207.237224] LNet: Added LNI 192.168.206.144@tcp133 [8/256/0/180] [13208.319458] LNet: Added LNI 192.168.206.144@tcp134 [8/256/0/180] [13209.157462] LNet: Added LNI 192.168.206.144@tcp135 [8/256/0/180] [13214.771304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13214.808280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13215.954075] Lustre: 463894:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.144@tcp: can't add NID 192.168.206.144@tcp130: rc = -75 [13216.061675] LNetError: 461352:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.144 [13216.080390] LNetError: Protocol error connecting to 192.168.206.144@tcp116 at host 192.168.206.144:988: is it running a compatible version of Lustre? [13216.091321] LNetError: Skipped 2 previous similar messages [13216.193756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13216.229432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13216.296364] Lustre: lustre-MDT0000: new disk, initializing [13216.401098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13216.426054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13216.735118] LNet: 5 local NIs in recovery (showing 5): 192.168.206.144@tcp124, 192.168.206.144@tcp126, 192.168.206.144@tcp127, 192.168.206.144@tcp129, 192.168.206.144@tcp128 [13217.120062] LNetError: 463917:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.144 [13217.129175] LNetError: 463917:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [13217.141914] LNetError: Protocol error connecting to 192.168.206.144@tcp106 at host 192.168.206.144:988: is it running a compatible version of Lustre? [13217.158155] LNetError: Skipped 7 previous similar messages [13219.809081] LNetError: 463914:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.144 [13219.818154] LNetError: 463914:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [13219.835176] LNetError: Protocol error connecting to 192.168.206.144@tcp124 at host 192.168.206.144:988: is it running a compatible version of Lustre? [13219.844828] LNetError: Skipped 5 previous similar messages [13220.373476] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [13223.906382] LNetError: 463927:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.144 [13223.915835] LNetError: Protocol error connecting to 192.168.206.144@tcp124 at host 192.168.206.144:988: is it running a compatible version of Lustre? [13229.054934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13229.210119] Lustre: 463910:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13229.275527] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13229.280455] Lustre: Skipped 1 previous similar message [13229.390239] Lustre: lustre-MDT0001: new disk, initializing [13229.445248] Lustre: 464711:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.144@tcp: can't add NID 192.168.206.144@tcp130: rc = -75 [13229.453800] Lustre: 464711:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13229.583729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13229.609255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13229.618047] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13231.076566] LNetError: 463923:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.144 [13231.094738] LNetError: Protocol error connecting to 192.168.206.144@tcp124 at host 192.168.206.144:988: is it running a compatible version of Lustre? [13233.614705] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [13241.950533] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13246.447796] LNetError: 461349:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.144 [13246.473762] LNetError: Protocol error connecting to 192.168.206.144@tcp117 at host 192.168.206.144:988: is it running a compatible version of Lustre? [13246.785291] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13252.872130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13253.126691] Lustre: lustre-OST0000: new disk, initializing [13253.131241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13253.135434] Lustre: 465884:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13253.206155] Lustre: 465889:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.144@tcp: can't add NID 192.168.206.144@tcp130: rc = -75 [13253.211690] Lustre: 465889:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [13253.247890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13259.737926] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [13261.394836] Lustre: 466317:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.144@tcp: can't add NID 192.168.206.144@tcp130: rc = -75 [13261.403925] Lustre: 466317:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13261.455931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13261.484252] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13261.620953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13263.909114] LNetError: 461349:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.144 [13263.924108] LNetError: 461349:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [13263.939738] LNetError: Protocol error connecting to 192.168.206.144@tcp123 at host 192.168.206.144:988: is it running a compatible version of Lustre? [13263.947444] LNetError: Skipped 9 previous similar messages [13270.090806] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13277.152728] LNet: 1 local NIs in recovery (showing 1): 192.168.206.144@tcp124 [13277.156050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13277.157214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13277.157728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13277.205108] Lustre: Skipped 1 previous similar message [13282.339660] Lustre: server umount lustre-OST0000 complete [13285.861121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13285.876237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13285.884218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13285.886741] Lustre: Skipped 1 previous similar message [13287.394768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13287.401487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13287.417146] Lustre: Skipped 1 previous similar message [13291.732559] Lustre: server umount lustre-MDT0000 complete [13296.003425] LustreError: 463903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841089 with bad export cookie 10651849794416632737 [13296.019514] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13296.033307] LustreError: 463903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13296.624760] Lustre: server umount lustre-MDT0001 complete [13296.685200] LNetError: 463927:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.144 [13296.712619] LNetError: 463927:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 17 previous similar messages [13296.726170] LNetError: Protocol error connecting to 192.168.206.144@tcp122 at host 192.168.206.144:988: is it running a compatible version of Lustre? [13296.753310] LNetError: Skipped 17 previous similar messages [13311.691241] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [13315.231042] Key type lgssc unregistered [13315.613300] LNet: 467800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13315.627434] LNetError: 467800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13315.637545] LNet: Removed LNI 192.168.206.144@tcp [13316.703171] LNet: Removed LNI 192.168.206.144@tcp100 [13316.707950] LNetError: 467800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13316.717982] LNetError: 467800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13316.738473] LNet: Removed LNI 192.168.206.144@tcp101 [13317.791151] LNet: Removed LNI 192.168.206.144@tcp102 [13317.825805] LNetError: 467800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13317.846495] LNetError: 467800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13317.858280] LNet: Removed LNI 192.168.206.144@tcp103 [13318.880730] LNet: Removed LNI 192.168.206.144@tcp104 [13319.903223] LNet: Removed LNI 192.168.206.144@tcp105 [13319.904853] LNetError: 467800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13319.910256] LNetError: 467800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13319.928097] LNet: Removed LNI 192.168.206.144@tcp106 [13320.991263] LNet: Removed LNI 192.168.206.144@tcp107 [13322.015159] LNet: Removed LNI 192.168.206.144@tcp108 [13323.039642] LNet: Removed LNI 192.168.206.144@tcp109 [13324.063242] LNet: Removed LNI 192.168.206.144@tcp110 [13324.065839] LNetError: 467800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13324.078864] LNetError: 467800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13324.089579] LNet: Removed LNI 192.168.206.144@tcp111 [13325.151386] LNet: Removed LNI 192.168.206.144@tcp112 [13326.175176] LNet: Removed LNI 192.168.206.144@tcp113 [13327.199173] LNet: Removed LNI 192.168.206.144@tcp114 [13328.223261] LNet: Removed LNI 192.168.206.144@tcp115 [13329.247198] LNet: Removed LNI 192.168.206.144@tcp116 [13330.271131] LNet: Removed LNI 192.168.206.144@tcp117 [13331.297381] LNet: Removed LNI 192.168.206.144@tcp118 [13332.319831] LNet: Removed LNI 192.168.206.144@tcp119 [13332.321801] LNetError: 467800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13332.331038] LNetError: 467800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13332.343605] LNet: Removed LNI 192.168.206.144@tcp120 [13333.409105] LNet: Removed LNI 192.168.206.144@tcp121 [13334.431686] LNet: Removed LNI 192.168.206.144@tcp122 [13335.455377] LNet: Removed LNI 192.168.206.144@tcp123 [13335.459530] LNet: Removed LNI 192.168.206.144@tcp124 [13336.479138] LNet: Removed LNI 192.168.206.144@tcp125 [13337.505165] LNet: Removed LNI 192.168.206.144@tcp126 [13338.527172] LNet: Removed LNI 192.168.206.144@tcp127 [13339.551166] LNet: Removed LNI 192.168.206.144@tcp128 [13340.575199] LNet: Removed LNI 192.168.206.144@tcp129 [13340.578146] LNet: Removed LNI 192.168.206.144@tcp130 [13340.585694] LNet: Removed LNI 192.168.206.144@tcp131 [13340.590054] LNet: Removed LNI 192.168.206.144@tcp132 [13340.595968] LNet: Removed LNI 192.168.206.144@tcp133 [13340.600325] LNet: Removed LNI 192.168.206.144@tcp134 [13340.638285] LNet: Removed LNI 192.168.206.144@tcp135 [13341.490333] Key type .llcrypt unregistered [13341.492833] Key type ._llcrypt unregistered [13362.321990] Key type ._llcrypt registered [13362.323410] Key type .llcrypt registered [13362.421286] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [13375.928777] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [13376.662441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13376.795351] alg: No test for adler32 (adler32-zlib) [13377.877282] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [13378.126255] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [13379.759266] Key type lgssc registered [13380.817406] Lustre: Echo OBD driver; http://www.lustre.org/ [13415.615899] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 10:33:27 (1787841207) [13432.663182] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [13436.469313] Key type lgssc unregistered [13436.985402] LNet: 471871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13437.000165] LNetError: 471871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13437.042920] LNet: Removed LNI 192.168.206.144@tcp [13438.287396] Key type .llcrypt unregistered [13438.293111] Key type ._llcrypt unregistered [13456.069212] Key type ._llcrypt registered [13456.071689] Key type .llcrypt registered [13456.182701] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [13457.077495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13457.157519] alg: No test for adler32 (adler32-zlib) [13458.306310] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [13458.616581] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [13460.448069] Key type lgssc registered [13462.111352] Lustre: Echo OBD driver; http://www.lustre.org/ [13472.737821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13472.768757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13474.145461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13474.187918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13474.259806] Lustre: lustre-MDT0000: new disk, initializing [13474.370903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13474.385820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13479.273916] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [13488.818348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13488.912275] Lustre: 473085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13488.960573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13488.964388] Lustre: Skipped 1 previous similar message [13489.039748] Lustre: lustre-MDT0001: new disk, initializing [13489.163750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13489.213412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13489.222537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13493.327798] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [13502.183972] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13507.991730] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13514.833274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13515.135316] Lustre: lustre-OST0000: new disk, initializing [13515.140375] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13515.154746] Lustre: 475056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13515.254657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13521.700865] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [13522.552232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13522.576949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13522.808474] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13532.631281] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13538.289050] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing lnet_if_list [13544.710775] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13544.763217] LNet: Added LNI 192.168.206.144@tcp10 [8/256/0/180] [13551.318569] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13551.365264] LNet: Added LNI 192.168.206.144@tcp11 [8/256/0/180] [13555.458589] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13555.491428] LNet: Added LNI 192.168.206.144@tcp12 [8/256/0/180] [13560.646510] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13560.680357] LNet: Added LNI 192.168.206.144@tcp13 [8/256/0/180] [13566.181313] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13566.229772] LNet: Added LNI 192.168.206.144@tcp14 [8/256/0/180] [13571.202989] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13571.239082] LNet: Added LNI 192.168.206.144@tcp15 [8/256/0/180] [13575.392906] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13575.432043] LNet: Added LNI 192.168.206.144@tcp16 [8/256/0/180] [13579.502434] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13579.524819] LNet: Added LNI 192.168.206.144@tcp17 [8/256/0/180] [13583.859373] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13583.884821] LNet: Added LNI 192.168.206.144@tcp18 [8/256/0/180] [13588.593533] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13588.629887] LNet: Added LNI 192.168.206.144@tcp19 [8/256/0/180] [13593.179486] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13593.208258] LNet: Added LNI 192.168.206.144@tcp20 [8/256/0/180] [13597.877372] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13597.912382] LNet: Added LNI 192.168.206.144@tcp21 [8/256/0/180] [13602.906594] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13602.930065] LNet: Added LNI 192.168.206.144@tcp22 [8/256/0/180] [13607.582532] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13607.624470] LNet: Added LNI 192.168.206.144@tcp23 [8/256/0/180] [13612.681888] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13612.710237] LNet: Added LNI 192.168.206.144@tcp24 [8/256/0/180] [13617.786634] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13617.832940] LNet: Added LNI 192.168.206.144@tcp25 [8/256/0/180] [13623.599467] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13623.630379] LNet: Added LNI 192.168.206.144@tcp26 [8/256/0/180] [13629.204287] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13629.237693] LNet: Added LNI 192.168.206.144@tcp27 [8/256/0/180] [13634.991710] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13635.066450] LNet: Added LNI 192.168.206.144@tcp28 [8/256/0/180] [13640.498893] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13640.544636] LNet: Added LNI 192.168.206.144@tcp29 [8/256/0/180] [13646.194745] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13646.214362] LNet: Added LNI 192.168.206.144@tcp30 [8/256/0/180] [13652.948604] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13652.978267] LNet: Added LNI 192.168.206.144@tcp31 [8/256/0/180] [13659.489744] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13659.513130] LNet: Added LNI 192.168.206.144@tcp32 [8/256/0/180] [13666.231541] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13666.260441] LNet: Added LNI 192.168.206.144@tcp33 [8/256/0/180] [13672.731278] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13672.771212] LNet: Added LNI 192.168.206.144@tcp34 [8/256/0/180] [13678.884299] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13678.930825] LNet: Added LNI 192.168.206.144@tcp35 [8/256/0/180] [13685.055685] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13685.092515] LNet: Added LNI 192.168.206.144@tcp36 [8/256/0/180] [13691.807523] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13691.828599] LNet: Added LNI 192.168.206.144@tcp37 [8/256/0/180] [13697.641597] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13697.669802] LNet: Added LNI 192.168.206.144@tcp38 [8/256/0/180] [13704.958512] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13705.010806] LNet: Added LNI 192.168.206.144@tcp39 [8/256/0/180] [13712.325945] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13712.354750] LNet: Added LNI 192.168.206.144@tcp40 [8/256/0/180] [13720.346561] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13720.382614] LNet: Added LNI 192.168.206.144@tcp41 [8/256/0/180] [13728.870138] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13728.937334] LNet: Added LNI 192.168.206.144@tcp42 [8/256/0/180] [13738.972382] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13739.043914] LNet: Added LNI 192.168.206.144@tcp43 [8/256/0/180] [13749.718340] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13749.762452] LNet: Added LNI 192.168.206.144@tcp44 [8/256/0/180] [13757.946396] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13757.969519] LNet: Added LNI 192.168.206.144@tcp45 [8/256/0/180] [13766.096767] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13766.118639] LNet: Added LNI 192.168.206.144@tcp46 [8/256/0/180] [13774.084371] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13774.115431] LNet: Added LNI 192.168.206.144@tcp47 [8/256/0/180] [13782.585173] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13782.615438] LNet: Added LNI 192.168.206.144@tcp48 [8/256/0/180] [13790.621142] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13790.640564] LNet: Added LNI 192.168.206.144@tcp49 [8/256/0/180] [13799.032535] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13799.062290] LNet: Added LNI 192.168.206.144@tcp50 [8/256/0/180] [13817.828062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13817.833185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13817.855264] Lustre: Skipped 1 previous similar message [13817.871463] Lustre: Skipped 2 previous similar messages [13822.944648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13822.957579] Lustre: Skipped 2 previous similar messages [13823.695464] Lustre: server umount lustre-MDT0000 complete [13827.263295] LustreError: 473076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841621 with bad export cookie 4955519831177274615 [13827.269454] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13827.278409] LustreError: 473076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13827.826991] Lustre: server umount lustre-MDT0001 complete [13843.427767] Lustre: 472357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841621/real 1787841621] req@000000001842a05e x1874687428822016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787841637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13843.451874] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13843.467022] Lustre: Skipped 1 previous similar message [13844.889130] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [13848.225171] Lustre: 472358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841625/real 1787841625] req@0000000034eab770 x1874687428822144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787841641 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13853.669107] Lustre: 472356:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841631/real 1787841631] req@0000000089f2b2e5 x1874687428822528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787841647 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13853.715586] Lustre: 472356:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [13853.825738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13858.783555] Lustre: 472357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841636/real 1787841636] req@000000004c68370f x1874687428822656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787841652 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13873.183524] LustreError: 472354:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000091cdd044 x1874687428823424/t0(0) o250->MGC192.168.206.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13873.519348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13877.975835] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [13878.568844] LustreError: 475050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13878.601578] LustreError: 475050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13883.880554] LustreError: 475051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13886.319798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13886.685851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13888.745984] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13890.992412] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [13897.973276] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13898.149909] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13903.830633] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13922.788095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13922.795222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13922.831956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13927.849462] Lustre: server umount lustre-OST0000 complete [13933.024565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13933.027049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13933.051446] Lustre: Skipped 1 previous similar message [13933.059324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13933.063985] Lustre: Skipped 2 previous similar messages [13939.168277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13939.176665] Lustre: Skipped 3 previous similar messages [13946.335399] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13946.583354] Lustre: server umount lustre-MDT0000 complete [13948.387550] LustreError: 499295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13948.405453] LustreError: 499295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13950.069600] LustreError: 473076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841743 with bad export cookie 4955519831177277135 [13950.073743] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13950.084949] LustreError: 473076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13950.385824] Lustre: server umount lustre-MDT0001 complete [13961.788591] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [13964.663693] Key type lgssc unregistered [13965.018707] LNet: 502078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13965.025350] LNetError: 502078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13966.047238] LNet: Removed LNI 192.168.206.144@tcp [13966.049229] LNetError: 502078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13966.056380] LNet: Removed LNI 192.168.206.144@tcp10 [13966.060508] LNet: Removed LNI 192.168.206.144@tcp11 [13966.063343] LNet: Removed LNI 192.168.206.144@tcp12 [13966.065850] LNet: Removed LNI 192.168.206.144@tcp13 [13966.068474] LNet: Removed LNI 192.168.206.144@tcp14 [13966.071171] LNet: Removed LNI 192.168.206.144@tcp15 [13966.073371] LNet: Removed LNI 192.168.206.144@tcp16 [13966.078762] LNet: Removed LNI 192.168.206.144@tcp17 [13966.086985] LNet: Removed LNI 192.168.206.144@tcp18 [13966.094487] LNet: Removed LNI 192.168.206.144@tcp19 [13966.100583] LNet: Removed LNI 192.168.206.144@tcp20 [13966.106637] LNet: Removed LNI 192.168.206.144@tcp21 [13966.109763] LNet: Removed LNI 192.168.206.144@tcp22 [13966.116463] LNet: Removed LNI 192.168.206.144@tcp23 [13966.122151] LNet: Removed LNI 192.168.206.144@tcp24 [13966.126267] LNet: Removed LNI 192.168.206.144@tcp25 [13966.128760] LNet: Removed LNI 192.168.206.144@tcp26 [13966.134918] LNet: Removed LNI 192.168.206.144@tcp27 [13966.142157] LNet: Removed LNI 192.168.206.144@tcp28 [13966.148529] LNet: Removed LNI 192.168.206.144@tcp29 [13966.154867] LNet: Removed LNI 192.168.206.144@tcp30 [13966.163949] LNet: Removed LNI 192.168.206.144@tcp31 [13966.167761] LNet: Removed LNI 192.168.206.144@tcp32 [13966.177780] LNet: Removed LNI 192.168.206.144@tcp33 [13966.182189] LNet: Removed LNI 192.168.206.144@tcp34 [13966.189287] LNet: Removed LNI 192.168.206.144@tcp35 [13966.192927] LNet: Removed LNI 192.168.206.144@tcp36 [13966.196416] LNet: Removed LNI 192.168.206.144@tcp37 [13966.200417] LNet: Removed LNI 192.168.206.144@tcp38 [13966.208952] LNet: Removed LNI 192.168.206.144@tcp39 [13966.214444] LNet: Removed LNI 192.168.206.144@tcp40 [13966.218305] LNet: Removed LNI 192.168.206.144@tcp41 [13966.225308] LNet: Removed LNI 192.168.206.144@tcp42 [13966.231470] LNet: Removed LNI 192.168.206.144@tcp43 [13966.234509] LNet: Removed LNI 192.168.206.144@tcp44 [13966.239863] LNet: Removed LNI 192.168.206.144@tcp45 [13966.247792] LNet: Removed LNI 192.168.206.144@tcp46 [13966.253816] LNet: Removed LNI 192.168.206.144@tcp47 [13966.261909] LNet: Removed LNI 192.168.206.144@tcp48 [13966.267525] LNet: Removed LNI 192.168.206.144@tcp49 [13966.276415] LNet: Removed LNI 192.168.206.144@tcp50 [13967.254906] Key type .llcrypt unregistered [13967.260255] Key type ._llcrypt unregistered [13989.970596] Key type ._llcrypt registered [13989.972996] Key type .llcrypt registered [13990.132964] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [14005.601948] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [14006.627898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14006.677035] alg: No test for adler32 (adler32-zlib) [14007.920851] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [14008.110476] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [14009.767295] Key type lgssc registered [14011.299109] Lustre: Echo OBD driver; http://www.lustre.org/ [14047.513976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14047.545456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14048.895069] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14048.928567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14049.021490] Lustre: lustre-MDT0000: new disk, initializing [14049.135933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14049.163602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14054.609868] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14065.853564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14065.927953] Lustre: 505276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14065.982381] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14065.989038] Lustre: Skipped 1 previous similar message [14066.047712] Lustre: lustre-MDT0001: new disk, initializing [14066.160499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14066.191695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14066.204816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14071.806880] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14083.519259] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14090.076718] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14098.455749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14098.595234] Lustre: *** cfs_fail_loc=90f, val=0*** [14098.722346] Lustre: lustre-OST0000: new disk, initializing [14098.725575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14098.732383] Lustre: 507292:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14098.851507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14105.750646] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14108.213382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14108.219645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14108.340257] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14116.428321] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14124.004176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14124.008184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14124.019267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14124.035373] Lustre: Skipped 1 previous similar message [14129.134646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14129.138559] Lustre: Skipped 1 previous similar message [14134.242931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14134.250099] Lustre: Skipped 1 previous similar message [14137.311382] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14137.465667] Lustre: server umount lustre-OST0000 complete [14142.945185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14142.957333] LustreError: Skipped 1 previous similar message [14142.971843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14142.993445] Lustre: Skipped 1 previous similar message [14142.999630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14144.481637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14149.601570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14149.609234] Lustre: Skipped 3 previous similar messages [14155.231430] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14155.483740] Lustre: server umount lustre-MDT0000 complete [14159.357890] LustreError: 505270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841953 with bad export cookie 2082328479842603035 [14159.358632] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14159.364476] LustreError: 505270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14159.623565] Lustre: server umount lustre-MDT0001 complete [14170.513861] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [14174.026109] Key type lgssc unregistered [14174.350756] LNet: 509205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14174.363305] LNetError: 509205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14174.383423] LNet: Removed LNI 192.168.206.144@tcp [14175.592394] Key type .llcrypt unregistered [14175.594059] Key type ._llcrypt unregistered [14197.016625] Key type ._llcrypt registered [14197.018132] Key type .llcrypt registered [14197.126636] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [14198.276859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14198.320024] alg: No test for adler32 (adler32-zlib) [14199.589294] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [14199.974891] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [14201.639193] Key type lgssc registered [14202.855553] Lustre: Echo OBD driver; http://www.lustre.org/ [14211.794079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14211.824947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14213.494576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14217.978489] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14223.842017] LustreError: 510478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14223.865156] LustreError: 510478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14227.075649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14227.603726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14231.991529] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14238.971741] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14243.855534] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14251.036744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14251.618863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14257.882974] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14265.670664] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14275.467482] loop0: detected capacity change from 0 to 204800000 [14275.686118] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14275.692836] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14275.703222] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14281.515645] loop0: detected capacity change from 0 to 204800000 [14281.759060] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14281.772403] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14281.789944] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14287.020657] loop0: detected capacity change from 0 to 204800000 [14293.070484] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14293.229338] Lustre: fs15246-OST0000: new disk, initializing [14293.233340] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14293.239398] Lustre: 514069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14293.315726] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14299.614134] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14310.009860] loop1: detected capacity change from 0 to 204800000 [14315.241234] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14315.378257] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14315.439653] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14315.534271] Lustre: fs15246-MDT0000: new disk, initializing [14315.679146] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14315.709202] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14315.795737] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14316.832162] Lustre: 515189:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000303738f0 x1874688205697792/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787842126 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14320.376557] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14325.926988] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14325.938032] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14325.963799] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14339.051942] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14339.063890] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14344.191570] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14344.944116] Lustre: server umount fs15246-MDT0000 complete [14349.289810] LustreError: 510472:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14352.663851] Lustre: server umount fs15246-OST0000 complete [14364.643345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14364.655690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14364.669908] Lustre: Skipped 1 previous similar message [14369.015598] Lustre: server umount lustre-OST0000 complete [14374.886545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14374.892904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14374.898687] Lustre: Skipped 1 previous similar message [14374.908515] Lustre: Skipped 1 previous similar message [14378.630377] Lustre: server umount lustre-MDT0000 complete [14380.002635] LustreError: 510474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14380.021913] LustreError: 510474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14381.920458] LustreError: 510458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842175 with bad export cookie 2563490723374224238 [14381.922809] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14381.934620] LustreError: 510458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14382.296754] Lustre: server umount lustre-MDT0001 complete [14392.984537] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [14395.744638] Key type lgssc unregistered [14396.087061] LNet: 518287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14396.093857] LNetError: 518287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14396.117212] LNet: Removed LNI 192.168.206.144@tcp [14396.858721] Key type .llcrypt unregistered [14396.860579] Key type ._llcrypt unregistered [14413.959992] Key type ._llcrypt registered [14413.962889] Key type .llcrypt registered [14414.100983] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [14415.076705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14415.208032] alg: No test for adler32 (adler32-zlib) [14416.265763] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [14416.589228] LNet: Added LNI 192.168.206.144@tcp [8/256/0/180] [14418.232321] Key type lgssc registered [14419.385613] Lustre: Echo OBD driver; http://www.lustre.org/ [14453.944397] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [14462.165178] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [14493.972194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14493.995111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14495.225037] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14495.252129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14495.337676] Lustre: lustre-MDT0000: new disk, initializing [14495.529498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14495.572464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14499.811594] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14509.845830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14509.958415] Lustre: 522751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14510.008974] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14510.013179] Lustre: Skipped 1 previous similar message [14510.099708] Lustre: lustre-MDT0001: new disk, initializing [14510.184921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14510.248731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14510.258968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14513.786614] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14521.955891] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14527.204763] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14533.613977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14533.874401] Lustre: lustre-OST0000: new disk, initializing [14533.880799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14533.907659] Lustre: 524723:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14534.038336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14539.978872] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_default_debug -1 all [14542.402338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14542.410700] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14542.519744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14549.366749] Lustre: DEBUG MARKER: oleg644-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14554.037549] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14554.319595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14558.910932] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14559.167806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14563.302705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14563.309108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14563.313585] LustreError: Skipped 1 previous similar message [14563.315670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14563.342824] Lustre: Skipped 1 previous similar message [14567.101441] Lustre: server umount lustre-OST0000 complete [14571.488474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14571.494865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14571.513883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14571.520842] Lustre: Skipped 1 previous similar message [14573.538437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14573.540167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14573.561340] Lustre: Skipped 1 previous similar message [14576.728564] Lustre: server umount lustre-MDT0000 complete [14580.079650] LustreError: 522744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842373 with bad export cookie 14167098060311273722 [14580.081787] LustreError: MGC192.168.206.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14580.095615] LustreError: 522744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14580.467789] Lustre: server umount lustre-MDT0001 complete [14604.306578] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing set_hostid [14614.275501] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing load_modules_local [14644.095373] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14363 sec ========= 10:53:54 (1787842434) [14646.456509] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:53:57 (1787842437) === [14650.725400] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:54:02 (1787842442) === [14692.045596] Lustre: DEBUG MARKER: oleg644-server.virtnet: executing unload_modules_local [14695.233710] Key type lgssc unregistered [14695.703889] LNet: 531366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14695.712905] LNetError: 531366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14695.743535] LNet: Removed LNI 192.168.206.144@tcp [14696.909623] Key type .llcrypt unregistered [14696.916560] Key type ._llcrypt unregistered