[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 484376171 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002008] Switched APIC routing to physical x2apic. [ 0.003012] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007010] pid_max: default: 32768 minimum: 301 [ 0.008120] LSM: Security Framework initializing [ 0.009039] Yama: becoming mindful. [ 0.010032] SELinux: Initializing. [ 0.011060] *** VALIDATE selinux *** [ 0.019171] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023652] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026118] *** VALIDATE tmpfs *** [ 0.028145] *** VALIDATE proc *** [ 0.029193] *** VALIDATE cgroup *** [ 0.030009] *** VALIDATE cgroup2 *** [ 0.031255] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032144] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035020] Spectre V2 : User space: Vulnerable [ 0.036007] Speculative Store Bypass: Vulnerable [ 0.038878] debug: unmapping init [mem 0xffffffff88659000-0xffffffff88660fff] [ 0.040159] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041754] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042027] ... version: 2 [ 0.043012] ... bit width: 48 [ 0.044011] ... generic registers: 4 [ 0.045012] ... value mask: 0000ffffffffffff [ 0.046013] ... max period: 00007fffffffffff [ 0.047014] ... fixed-purpose events: 3 [ 0.048013] ... event mask: 000000070000000f [ 0.049285] rcu: Hierarchical SRCU implementation. [ 0.051452] smp: Bringing up secondary CPUs ... [ 0.052602] x86: Booting SMP configuration: [ 0.053023] .... node #0, CPUs: #1 #2 #3 [ 0.057029] smp: Brought up 1 node, 4 CPUs [ 0.059020] smpboot: Max logical packages: 1 [ 0.060015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142023] node 0 deferred pages initialised in 78ms [ 0.145511] devtmpfs: initialized [ 0.146216] x86/mm: Memory block size: 128MB [ 0.148638] gcov: version magic: 0x41383552 [ 0.149686] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156460] pinctrl core: initialized pinctrl subsystem [ 0.158208] [ 0.158895] ************************************************************* [ 0.162014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164014] ** ** [ 0.167015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.169012] ** ** [ 0.172012] ** This means that this kernel is built to expose internal ** [ 0.174018] ** IOMMU data structures, which may compromise security on ** [ 0.176012] ** your system. ** [ 0.178012] ** ** [ 0.180012] ** If you see this message and you are not debugging the ** [ 0.183013] ** kernel, report this immediately to your vendor! ** [ 0.185064] ** ** [ 0.187013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189015] ************************************************************* [ 0.192825] NET: Registered protocol family 16 [ 0.194530] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.199071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203100] cpuidle: using governor menu [ 0.205931] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.208502] PCI: Using configuration type 1 for base access [ 0.211254] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.222064] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.224033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227088] cryptd: max_cpu_qlen set to 1000 [ 0.229831] ACPI: Added _OSI(Module Device) [ 0.230000] ACPI: Added _OSI(Processor Device) [ 0.233015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.235013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.240541] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246345] ACPI: Interpreter enabled [ 0.247059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.249010] ACPI: Using IOAPIC for interrupt routing [ 0.250133] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.253400] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.264191] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.267045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.269023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.273097] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.276373] acpiphp: Slot [2] registered [ 0.278096] acpiphp: Slot [5] registered [ 0.279104] acpiphp: Slot [6] registered [ 0.280075] acpiphp: Slot [7] registered [ 0.281088] acpiphp: Slot [8] registered [ 0.282115] acpiphp: Slot [9] registered [ 0.283100] acpiphp: Slot [10] registered [ 0.284117] acpiphp: Slot [3] registered [ 0.286111] acpiphp: Slot [4] registered [ 0.287082] acpiphp: Slot [11] registered [ 0.289086] acpiphp: Slot [12] registered [ 0.290094] acpiphp: Slot [13] registered [ 0.291119] acpiphp: Slot [14] registered [ 0.292088] acpiphp: Slot [15] registered [ 0.294114] acpiphp: Slot [16] registered [ 0.295102] acpiphp: Slot [17] registered [ 0.296094] acpiphp: Slot [18] registered [ 0.298090] acpiphp: Slot [19] registered [ 0.299125] acpiphp: Slot [20] registered [ 0.300125] acpiphp: Slot [21] registered [ 0.302080] acpiphp: Slot [22] registered [ 0.303099] acpiphp: Slot [23] registered [ 0.304090] acpiphp: Slot [24] registered [ 0.305093] acpiphp: Slot [25] registered [ 0.307108] acpiphp: Slot [26] registered [ 0.308118] acpiphp: Slot [27] registered [ 0.310183] acpiphp: Slot [28] registered [ 0.312167] acpiphp: Slot [29] registered [ 0.314161] acpiphp: Slot [30] registered [ 0.316170] acpiphp: Slot [31] registered [ 0.318094] PCI host bridge to bus 0000:00 [ 0.319034] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.325059] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.328034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.330024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.333031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.335188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.339027] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.342340] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.353015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.359033] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.362016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.364016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.366022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.369751] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.373799] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.376059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.379935] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.386029] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.403026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.408023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.415188] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.424019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.432018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.452018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.464049] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.475028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.486026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.505030] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.517509] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.524016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.530022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.544016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.554700] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.562023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.570024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.593021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.604020] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.611020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.620038] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.638018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.648339] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.660015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.667016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.690016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.705646] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.708576] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.711665] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.714592] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.717375] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.723155] iommu: Default domain type: Passthrough [ 0.725546] SCSI subsystem initialized [ 0.727213] ACPI: bus type USB registered [ 0.728127] usbcore: registered new interface driver usbfs [ 0.730093] usbcore: registered new interface driver hub [ 0.732081] usbcore: registered new device driver usb [ 0.734188] pps_core: LinuxPPS API ver. 1 registered [ 0.736013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.740067] PTP clock support registered [ 0.742138] EDAC MC: Ver: 3.0.0 [ 0.744135] PCI: Using ACPI for IRQ routing [ 0.746970] NetLabel: Initializing [ 0.748011] NetLabel: domain hash size = 128 [ 0.749009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.751097] NetLabel: unlabeled traffic allowed by default [ 0.753097] vgaarb: loaded [ 0.754254] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.756018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.762004] clocksource: Switched to clocksource kvm-clock [ 0.870923] VFS: Disk quotas dquot_6.6.0 [ 0.872762] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.875352] *** VALIDATE ramfs *** [ 0.876616] *** VALIDATE hugetlbfs *** [ 0.879076] pnp: PnP ACPI init [ 0.881510] pnp: PnP ACPI: found 6 devices [ 0.899600] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.902916] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.904712] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.906913] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.909301] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.911234] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.913726] NET: Registered protocol family 2 [ 0.916199] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.920703] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.924408] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.930294] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.935191] TCP: Hash tables configured (established 65536 bind 65536) [ 0.938985] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.942644] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.946166] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.949485] NET: Registered protocol family 1 [ 0.952570] RPC: Registered named UNIX socket transport module. [ 0.954732] RPC: Registered udp transport module. [ 0.956502] RPC: Registered tcp transport module. [ 0.958073] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.960435] NET: Registered protocol family 44 [ 0.962205] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.964276] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.966435] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.968495] PCI: CLS 0 bytes, default 64 [ 0.970223] Unpacking initramfs... [ 2.457989] debug: unmapping init [mem 0xffff89633cc54000-0xffff89633ffbffff] [ 2.464531] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.466582] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.470398] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.983826] Initialise system trusted keyrings [ 2.985681] Key type blacklist registered [ 3.006066] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.021341] zbud: loaded [ 3.024956] *** VALIDATE nfs *** [ 3.026045] *** VALIDATE nfs4 *** [ 3.027608] pstore: using deflate compression [ 3.031059] Platform Keyring initialized [ 3.141750] NET: Registered protocol family 38 [ 3.143574] Key type asymmetric registered [ 3.146683] Asymmetric key parser 'x509' registered [ 3.148247] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.150794] io scheduler mq-deadline registered [ 3.152262] io scheduler kyber registered [ 3.153615] io scheduler bfq registered [ 3.155407] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.158142] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.161389] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.164671] ACPI: Power Button [PWRF] [ 3.170602] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.177521] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.195965] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.203968] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.222698] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.250590] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.277637] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.283285] Non-volatile memory driver v1.3 [ 3.285590] Linux agpgart interface v0.103 [ 3.326794] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.329874] vda: detected capacity change from 0 to 74760192 [ 3.347584] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.350800] vdb: detected capacity change from 0 to 1073741824 [ 3.366862] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.369936] vdc: detected capacity change from 0 to 2621440000 [ 3.388484] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.391876] vdd: detected capacity change from 0 to 2621440000 [ 3.420351] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.423676] vde: detected capacity change from 0 to 4294967296 [ 3.442095] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.445663] vdf: detected capacity change from 0 to 4294967296 [ 3.455324] libphy: Fixed MDIO Bus: probed [ 3.462353] usbcore: registered new interface driver usbserial_generic [ 3.465310] usbserial: USB Serial support registered for generic [ 3.468023] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.472911] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.475119] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.478227] mousedev: PS/2 mouse device common for all mice [ 3.481800] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.484886] rtc_cmos 00:05: RTC can wake from S4 [ 3.491358] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.493363] rtc_cmos 00:05: registered as rtc0 [ 3.496438] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.497843] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.498707] intel_pstate: CPU model not supported [ 3.506919] hid: raw HID events driver (C) Jiri Kosina [ 3.509364] usbcore: registered new interface driver usbhid [ 3.511370] usbhid: USB HID core driver [ 3.513254] drop_monitor: Initializing network drop monitor service [ 3.515865] Initializing XFRM netlink socket [ 3.518197] NET: Registered protocol family 10 [ 3.523226] Segment Routing with IPv6 [ 3.524372] NET: Registered protocol family 17 [ 3.526288] mpls_gso: MPLS GSO support [ 3.531518] RAS: Correctable Errors collector initialized. [ 3.533264] AVX version of gcm_enc/dec engaged. [ 3.534475] AES CTR mode by8 optimization enabled [ 3.616071] sched_clock: Marking stable (3616010130, 0)->(4479636435, -863626305) [ 3.619964] registered taskstats version 1 [ 3.622428] Loading compiled-in X.509 certificates [ 3.624683] zswap: loaded using pool lzo/zbud [ 3.652873] Key type big_key registered [ 3.668094] Key type encrypted registered [ 3.669928] ima: No TPM chip found, activating TPM-bypass! [ 3.672120] ima: Allocated hash algorithm: sha1 [ 3.674036] ima: No architecture policies found [ 3.675797] evm: Initialising EVM extended attributes: [ 3.677721] evm: security.selinux [ 3.678992] evm: security.ima [ 3.680117] evm: security.capability [ 3.681548] evm: HMAC attrs: 0x1 [ 3.684704] rtc_cmos 00:05: setting system clock to 2026-08-31 04:50:36 UTC (1788151836) [ 3.691936] debug: unmapping init [mem 0xffffffff89603000-0xffffffff897fffff] [ 3.695368] debug: unmapping init [mem 0xffffffff88382000-0xffffffff88658fff] [ 3.705095] Write protecting the kernel read-only data: 28672k [ 3.708658] debug: unmapping init [mem 0xffffffff86a03000-0xffffffff86bfffff] [ 3.711973] debug: unmapping init [mem 0xffffffff87314000-0xffffffff873fffff] [ 3.755145] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.763151] systemd[1]: Detected virtualization kvm. [ 3.765529] systemd[1]: Detected architecture x86-64. [ 3.767696] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.795343] systemd[1]: No hostname configured. [ 3.797342] systemd[1]: Set hostname to . [ 3.799580] random: systemd: uninitialized urandom read (16 bytes read) [ 3.802385] systemd[1]: Initializing machine ID from random generator. [ 3.853862] random: ln: uninitialized urandom read (6 bytes read) [ 3.953880] random: systemd: uninitialized urandom read (16 bytes read) [ 3.957573] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.964229] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.969130] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.623229] device-mapper: uevent: version 1.0.3 [ 4.625830] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.438348] random: fast init done [ 5.454488] virtio_net virtio0 ens2: renamed from eth0 [ 5.473800] scsi host0: ata_piix [ 5.502123] scsi host1: ata_piix [ 5.503872] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.507759] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.982273] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.187243] random: crng init done [ 10.188741] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.731055] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.913965] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.214902] SELinux: Disabled at runtime. [ 12.277434] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.289832] systemd[1]: Detected virtualization kvm. [ 12.292382] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.851923] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.855818] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.861289] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.866728] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.871358] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.881552] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.885978] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ 13.071351] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.315948] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.700155] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.705001] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.892185] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.906450] EDAC sbridge: Ver: 1.1.2 [ 15.693771] Key type dns_resolver registered [ 16.008049] NFS: Registering the id_resolver key type [ 16.010086] Key type id_resolver registered [ 16.011796] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg425-server login: [ 37.472851] hrtimer: interrupt took 6566990 ns [ 64.335893] libcfs: loading out-of-tree module taints kernel. [ 64.420891] Key type ._llcrypt registered [ 64.424752] Key type .llcrypt registered [ 64.564885] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 85.892856] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 87.292092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 87.313520] alg: No test for adler32 (adler32-zlib) [ 88.821387] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 90.197143] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 92.023251] Key type lgssc registered [ 94.287365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 114.411169] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 157.545946] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 170.027300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 170.076410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.478288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 171.506031] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 171.574267] Lustre: lustre-MDT0000: new disk, initializing [ 171.655071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.670945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 178.861204] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.187846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 197.506297] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 197.572712] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 197.577298] Lustre: Skipped 1 previous similar message [ 197.652389] Lustre: lustre-MDT0001: new disk, initializing [ 197.907331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 197.982115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 197.994348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 203.322711] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.696046] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 220.094453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.464723] Lustre: lustre-OST0000: new disk, initializing [ 220.469816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 220.482327] Lustre: 8448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.589991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 226.934676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 226.951418] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 227.074863] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 228.356348] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.637507] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 244.828574] Lustre: lustre-OST0001: new disk, initializing [ 244.834729] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 244.846698] Lustre: 9524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 244.945326] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 251.936940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 251.948907] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 252.011362] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 252.049580] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 264.795339] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 273.699102] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 280.964824] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing check_logdir /tmp/testlogs/ [ 286.236199] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing yml_node [ 292.265546] Lustre: DEBUG MARKER: Client: 2.17.55.27 [ 295.067948] Lustre: DEBUG MARKER: MDS: 2.17.55.27 [ 298.096224] Lustre: DEBUG MARKER: OSS: 2.17.55.27 [ 300.278370] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 31 00:55:29 EDT 2026 [ 320.888546] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 323.138375] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 341.475236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 341.478753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 341.485547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 344.034511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.035039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 344.061522] Lustre: Skipped 2 previous similar messages [ 349.161250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 349.182628] Lustre: Skipped 3 previous similar messages [ 354.272520] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 354.280584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.308552] Lustre: Skipped 3 previous similar messages [ 354.861977] Lustre: server umount lustre-MDT0000 complete [ 359.649855] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152192 with bad export cookie 13080767696845172121 [ 359.650651] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 359.663656] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 364.512929] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.515037] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 364.525112] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 364.550614] Lustre: Skipped 3 previous similar messages [ 364.596501] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 366.226604] Lustre: server umount lustre-MDT0001 complete [ 370.410901] Lustre: server umount lustre-OST0000 complete [ 375.185395] Lustre: server umount lustre-OST0001 complete [ 383.227273] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 395.156851] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 437.021692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.256550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 437.280919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 437.369963] Lustre: lustre-MDT0000: new disk, initializing [ 437.475085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.498652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 442.024765] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 453.553501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 453.711403] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 453.816575] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 453.829316] Lustre: Skipped 1 previous similar message [ 453.975550] Lustre: lustre-MDT0001: new disk, initializing [ 454.147386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 454.190212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 454.208888] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 459.804228] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 470.130524] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 478.212859] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 486.451350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 486.724639] Lustre: lustre-OST0000: new disk, initializing [ 486.727981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 486.733858] Lustre: 17050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 486.786713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 488.893301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 488.925847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 489.165345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 494.435577] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 504.900159] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 510.103612] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 510.347435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 515.648353] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 515.848842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 519.649481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 519.658579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.675966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 519.682546] Lustre: Skipped 1 previous similar message [ 524.565468] Lustre: server umount lustre-OST0000 complete [ 530.401406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.415186] Lustre: Skipped 2 previous similar messages [ 540.645636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 540.660109] Lustre: Skipped 7 previous similar messages [ 543.199167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 543.454238] Lustre: server umount lustre-MDT0000 complete [ 545.760485] LustreError: 16500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.794828] LustreError: 16500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 547.716420] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152380 with bad export cookie 13080767696845173654 [ 547.724788] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 547.734839] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 548.051247] Lustre: server umount lustre-MDT0001 complete [ 552.829469] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 00:59:42 (1788152382) [ 555.535670] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 557.953274] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 00:59:47 (1788152387) [ 565.620471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.990908] LustreError: 19459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.073854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 570.984364] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 571.363703] LustreError: 19459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.377302] LustreError: 19459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 575.457667] LustreError: 19460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.808817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.092160] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 592.949467] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 599.553752] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 609.063586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 616.275496] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 626.257138] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 636.384611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 636.404557] LustreError: Skipped 1 previous similar message [ 636.407946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.421184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 640.497553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 641.752124] Lustre: server umount lustre-OST0000 complete [ 646.624745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 652.843742] Lustre: server umount lustre-MDT0000 complete [ 655.843847] LustreError: 21580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 655.868801] LustreError: 21580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 656.620113] LustreError: 20194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152489 with bad export cookie 13080767696845174529 [ 656.625669] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.638343] LustreError: 20194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 656.957701] Lustre: server umount lustre-MDT0001 complete [ 668.999411] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 672.012516] Key type lgssc unregistered [ 672.350369] LNet: 23008:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.354467] LNetError: 23008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 672.370641] LNet: Removed LNI 192.168.204.125@tcp [ 673.389270] Key type .llcrypt unregistered [ 673.392548] Key type ._llcrypt unregistered [ 700.663774] Key type ._llcrypt registered [ 700.672151] Key type .llcrypt registered [ 700.843303] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 702.139700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 702.190251] alg: No test for adler32 (adler32-zlib) [ 703.235801] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 703.545125] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 705.328251] Key type lgssc registered [ 706.712591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 716.120131] loop: module loaded [ 716.217098] loop0: detected capacity change from 0 to 204800000 [ 716.395866] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 716.414470] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 716.808741] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 723.526882] loop0: detected capacity change from 0 to 204800000 [ 723.817505] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 723.832875] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 723.848060] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 729.169963] loop0: detected capacity change from 0 to 204800000 [ 734.245946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 734.269683] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 735.477385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 735.538979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 735.597771] Lustre: lustre-MDT0000: new disk, initializing [ 735.678394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 735.703156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 740.425411] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 749.952314] loop1: detected capacity change from 0 to 204800000 [ 756.581548] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 756.853194] Lustre: lustre-OST07c6: new disk, initializing [ 756.858893] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 756.863974] Lustre: Skipped 1 previous similar message [ 756.870893] Lustre: 26198:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 757.008641] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 764.245658] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 766.597951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 766.611842] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 766.669473] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 771.014552] Lustre: 25182:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 771.025120] Lustre: 25182:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 771.036971] Lustre: 25182:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 771.052172] Lustre: 25182:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 771.060973] Lustre: 25182:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 771.079663] Lustre: 25182:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 786.345469] Lustre: 25182:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 786.353549] Lustre: 25182:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 786.357554] Lustre: 25182:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 786.367140] Lustre: 25182:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 786.373558] Lustre: 25182:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 786.380594] Lustre: 25182:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 791.549990] Lustre: 26723:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 791.560274] Lustre: 26723:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 791.569121] Lustre: 26723:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 791.575411] Lustre: 26723:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.584502] Lustre: 26723:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 791.589948] Lustre: 26723:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.595564] Lustre: 26723:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 791.601169] Lustre: 26723:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.606785] Lustre: 26723:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 791.611513] Lustre: 26723:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.620594] Lustre: 26723:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 791.626371] Lustre: 26723:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.809980] Lustre: 25183:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 794.822919] Lustre: 25183:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 794.833324] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 794.839284] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.846448] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 794.852187] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.857583] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 794.862611] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.868251] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 794.872519] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.881901] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 794.891042] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.663543] Lustre: 25184:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 799.673150] Lustre: 25184:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 799.696377] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 799.706285] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.720948] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 799.730349] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.739283] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 799.749711] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.758284] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 799.763385] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.775415] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 799.781216] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 807.844310] Lustre: 25183:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 807.858868] Lustre: 25183:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 807.864951] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 807.869126] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 807.873959] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 807.878861] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 807.883814] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 807.891257] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 807.909192] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 807.915687] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 807.921556] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 807.936330] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 857.724171] Lustre: server umount lustre-OST07c6 complete [ 876.646851] Lustre: server umount lustre-MDT0000 complete [ 895.173623] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 898.069460] Key type lgssc unregistered [ 898.460503] LNet: 28751:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.468647] LNetError: 28751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.486958] LNet: Removed LNI 192.168.204.125@tcp [ 899.316653] Key type .llcrypt unregistered [ 899.318683] Key type ._llcrypt unregistered [ 918.333213] Key type ._llcrypt registered [ 918.335962] Key type .llcrypt registered [ 918.461944] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 919.572174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 919.743231] alg: No test for adler32 (adler32-zlib) [ 920.940828] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 921.246128] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 922.930540] Key type lgssc registered [ 923.851188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 930.272382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 930.304941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.695692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 935.078763] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 942.068961] LustreError: 30023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 942.095687] LustreError: 30023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 943.443489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.815307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 948.840084] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 957.266275] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 963.794328] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 970.551734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 970.874363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 976.839696] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 977.021918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 984.282853] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1004.005117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1004.006710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.029314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1004.048306] Lustre: Skipped 1 previous similar message [ 1005.351496] Lustre: server umount lustre-OST0000 complete [ 1010.659339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1010.668622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.694621] Lustre: Skipped 1 previous similar message [ 1010.704384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1014.241425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1014.241895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.549301] Lustre: server umount lustre-MDT0000 complete [ 1019.362248] LustreError: 30019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1019.379276] LustreError: 30019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1019.500844] LustreError: 30003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152852 with bad export cookie 2590106332710198849 [ 1019.520930] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1019.871302] Lustre: server umount lustre-MDT0001 complete [ 1031.351281] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1033.912723] Key type lgssc unregistered [ 1034.169813] LNet: 33761:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.181237] LNetError: 33761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.244989] LNet: Removed LNI 192.168.204.125@tcp [ 1036.066394] Key type .llcrypt unregistered [ 1036.068457] Key type ._llcrypt unregistered [ 1052.610553] Key type ._llcrypt registered [ 1052.612754] Key type .llcrypt registered [ 1052.704918] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1053.564487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1053.597590] alg: No test for adler32 (adler32-zlib) [ 1054.572956] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1054.787047] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1056.447249] Key type lgssc registered [ 1057.721804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1064.912286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1064.924874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.270947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.962212] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1076.707631] LustreError: 35033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1076.735863] LustreError: 35033:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1077.010522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1077.440784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1081.860995] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1088.659880] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1093.485793] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1099.398872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1099.698948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1104.529858] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1110.371968] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1111.990652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1127.394858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.395078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1127.398234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1127.403625] Lustre: Skipped 1 previous similar message [ 1130.495445] Lustre: server umount lustre-OST0000 complete [ 1134.048589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1134.060846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.078300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1134.085649] Lustre: Skipped 1 previous similar message [ 1137.633205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1137.633834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.752894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1142.764749] Lustre: Skipped 2 previous similar messages [ 1147.879646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1148.383214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1148.638083] Lustre: server umount lustre-MDT0000 complete [ 1152.329882] LustreError: 35015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152985 with bad export cookie 6734928296469170306 [ 1152.332821] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1152.344352] LustreError: 35015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1152.698264] Lustre: server umount lustre-MDT0001 complete [ 1161.349531] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1163.759405] Key type lgssc unregistered [ 1163.980098] LNet: 38585:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.990961] LNetError: 38585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1164.011888] LNet: Removed LNI 192.168.204.125@tcp [ 1164.846116] Key type .llcrypt unregistered [ 1164.849627] Key type ._llcrypt unregistered [ 1184.758442] Key type ._llcrypt registered [ 1184.761621] Key type .llcrypt registered [ 1184.892514] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1185.752770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1185.785407] alg: No test for adler32 (adler32-zlib) [ 1186.789553] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1187.046863] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1188.735411] Key type lgssc registered [ 1189.770045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1196.687338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1196.704327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.227902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1202.209312] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1207.782130] LustreError: 39980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1207.809683] LustreError: 39980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1210.306078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1210.630549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1214.746811] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1222.568666] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1227.674051] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1233.655313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1233.975226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1238.854713] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1243.201777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1246.045403] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1253.358652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.359512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1253.377252] Lustre: Skipped 2 previous similar messages [ 1253.386019] Lustre: Skipped 2 previous similar messages [ 1253.976618] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.25@tcp (stopping) [ 1254.369929] Lustre: server umount lustre-MDT0000 complete [ 1258.316408] LustreError: 39961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153091 with bad export cookie 4570724351051839305 [ 1258.319019] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1258.323986] LustreError: 39961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1258.465111] LustreError: 39975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1258.470305] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.491414] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1263.584446] LustreError: 41220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.584876] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1263.616534] LustreError: 41220:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1263.641728] Lustre: Skipped 1 previous similar message [ 1264.744805] Lustre: server umount lustre-MDT0001 complete [ 1286.449886] Lustre: server umount lustre-OST0000 complete [ 1301.801415] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1304.342485] Key type lgssc unregistered [ 1304.557298] LNet: 43840:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1304.563224] LNetError: 43840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1304.576482] LNet: Removed LNI 192.168.204.125@tcp [ 1305.247834] Key type .llcrypt unregistered [ 1305.252883] Key type ._llcrypt unregistered [ 1329.456255] Key type ._llcrypt registered [ 1329.458030] Key type .llcrypt registered [ 1329.513158] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1330.517659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1330.604957] alg: No test for adler32 (adler32-zlib) [ 1331.608721] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1331.913768] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1333.591531] Key type lgssc registered [ 1334.739451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1342.228706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1342.257758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1343.629861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1348.108922] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1353.250981] LustreError: 45256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1353.266405] LustreError: 45256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1358.312371] LustreError: 45255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.056220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1359.535790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1364.292700] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1373.440628] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1379.993880] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1387.527690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1387.874648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1388.929221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1395.693247] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1403.656181] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1409.503996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1409.507053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1409.516689] LustreError: Skipped 1 previous similar message [ 1409.519522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1409.548119] Lustre: Skipped 1 previous similar message [ 1410.671817] Lustre: lustre-OST0000: Not available for connect from 192.168.204.25@tcp (stopping) [ 1410.682217] Lustre: Skipped 1 previous similar message [ 1414.631366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1414.635142] Lustre: Skipped 1 previous similar message [ 1419.749645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1419.760944] Lustre: Skipped 2 previous similar messages [ 1423.327656] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1423.535542] Lustre: server umount lustre-OST0000 complete [ 1443.120064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.136449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1443.143904] Lustre: Skipped 2 previous similar messages [ 1449.442518] LustreError: 45986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.458504] LustreError: 45986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1449.532330] Lustre: server umount lustre-MDT0000 complete [ 1454.567431] LustreError: 45255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1454.602134] LustreError: 45255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1455.408114] LustreError: 45993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153288 with bad export cookie 2759800128008022941 [ 1455.413923] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1455.431064] LustreError: 45993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1462.136526] Lustre: server umount lustre-MDT0001 complete [ 1476.593275] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1479.894713] Key type lgssc unregistered [ 1480.256080] LNet: 48965:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.265213] LNetError: 48965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.279606] LNet: Removed LNI 192.168.204.125@tcp [ 1481.114598] Key type .llcrypt unregistered [ 1481.127666] Key type ._llcrypt unregistered [ 1505.547498] Key type ._llcrypt registered [ 1505.550643] Key type .llcrypt registered [ 1505.656733] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1506.850424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1506.929311] alg: No test for adler32 (adler32-zlib) [ 1508.158126] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1508.527531] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1510.247277] Key type lgssc registered [ 1511.325278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1518.452125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1518.475252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1519.935701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1524.397288] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1529.315501] LustreError: 50400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1529.337188] LustreError: 50400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1533.193676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1533.569562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1538.027437] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1545.380045] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1551.458868] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1559.954613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1560.240619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1565.379265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1567.866163] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1576.519236] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1583.484069] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1587.943780] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1601.016700] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-31 1h17m09s [ 1603.142915] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1609.697798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.709211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1609.721556] Lustre: Skipped 2 previous similar messages [ 1609.733849] Lustre: Skipped 2 previous similar messages [ 1610.787612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1610.793897] Lustre: Skipped 3 previous similar messages [ 1611.861906] Lustre: server umount lustre-MDT0000 complete [ 1615.951902] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1616.869168] LustreError: 50401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1616.904182] LustreError: 50401:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1623.015761] LustreError: 51128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1623.040238] LustreError: 51128:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1623.460981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1623.606707] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1623.824176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1623.888817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1628.398288] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1630.198395] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1632.258357] Lustre: DEBUG MARKER: Wait for df (36730) ... [ 1633.824887] Lustre: DEBUG MARKER: done [ 1639.397798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1639.407788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.420343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1639.427660] Lustre: Skipped 1 previous similar message [ 1640.423812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1640.432779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.557485] Lustre: server umount lustre-OST0000 complete [ 1650.658240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1650.658475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1650.660919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1650.673979] Lustre: Skipped 1 previous similar message [ 1650.713804] Lustre: Skipped 3 previous similar messages [ 1654.857158] Lustre: server umount lustre-MDT0000 complete [ 1657.829001] LustreError: 51128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1657.837043] LustreError: 51128:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1658.519337] LustreError: 50382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153491 with bad export cookie 15479441207402481206 [ 1658.524713] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1658.531315] LustreError: 50382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1658.855390] Lustre: server umount lustre-MDT0001 complete [ 1669.419534] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1672.139114] Key type lgssc unregistered [ 1672.458201] LNet: 55328:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.465693] LNetError: 55328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1672.494156] LNet: Removed LNI 192.168.204.125@tcp [ 1673.283455] Key type .llcrypt unregistered [ 1673.286294] Key type ._llcrypt unregistered [ 1713.186603] Key type ._llcrypt registered [ 1713.188470] Key type .llcrypt registered [ 1713.276782] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1714.344402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1714.370660] alg: No test for adler32 (adler32-zlib) [ 1715.528157] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1715.886110] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1717.623161] Key type lgssc registered [ 1718.871942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1727.279306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1727.300394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1728.649677] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1728.664589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1728.811088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1733.442307] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1742.875276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1743.044177] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1743.108763] Lustre: 57534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1743.387118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1748.161369] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1755.888184] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1761.317120] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1767.897899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1768.052799] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1768.200957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1773.479805] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1775.146962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1780.814685] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1788.772900] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-31 1h20m18s [ 1790.885456] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1798.146966] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1801.058448] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1801.156604] LustreError: 57541:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1821.295762] Lustre: lustre-MDT0000: Client 392560b9-7dd0-4cfa-a116-589d48f2aa1e (at 192.168.204.25@tcp) reconnecting [ 1841.255481] LustreError: 57541:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1843.460090] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1845.623637] Lustre: DEBUG MARKER: done [ 1851.877133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1851.879052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1851.892897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1851.894020] LustreError: Skipped 1 previous similar message [ 1851.911489] Lustre: Skipped 1 previous similar message [ 1856.871816] Lustre: server umount lustre-OST0000 complete [ 1861.088904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1861.107071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.122959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1861.134551] Lustre: Skipped 1 previous similar message [ 1862.186810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1862.198353] Lustre: Skipped 2 previous similar messages [ 1866.519266] Lustre: server umount lustre-MDT0000 complete [ 1870.400947] LustreError: 57528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153703 with bad export cookie 8790931074512839019 [ 1870.408120] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1870.413203] LustreError: 57528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1870.683759] Lustre: server umount lustre-MDT0001 complete [ 1880.784865] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1883.444610] Key type lgssc unregistered [ 1883.729159] LNet: 61872:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.749714] LNetError: 61872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1883.776195] LNet: Removed LNI 192.168.204.125@tcp [ 1884.729087] Key type .llcrypt unregistered [ 1884.730772] Key type ._llcrypt unregistered [ 1927.646550] Key type ._llcrypt registered [ 1927.648666] Key type .llcrypt registered [ 1927.748595] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1928.835584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1928.879984] alg: No test for adler32 (adler32-zlib) [ 1929.909530] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1930.162454] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1931.911411] Key type lgssc registered [ 1932.972763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1936.069411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1936.094034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1937.343843] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1937.367935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1937.515287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.326641] Lustre: Failing over lustre-MDT0000 [ 1938.557503] Lustre: server umount lustre-MDT0000 complete [ 1945.002791] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:22:55 (1788153775) [ 1951.202597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.576392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.262115] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1962.489730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.580927] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1962.601411] Lustre: 64672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1962.775499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1966.122610] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1972.528871] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1978.425951] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1984.584976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1984.799762] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1985.035588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1991.787571] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1993.351138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1998.510608] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2004.352593] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2009.810462] Lustre: server umount lustre-MDT0000 complete [ 2012.453945] LustreError: 64666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153845 with bad export cookie 7377228225567398949 [ 2012.462437] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2012.474719] LustreError: 64666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2014.176664] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.179786] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2014.185132] Lustre: Skipped 1 previous similar message [ 2014.717135] Lustre: server umount lustre-MDT0001 complete [ 2017.618793] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2022.310357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2030.468424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.664197] LustreError: 63594:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000765c46ae x1875015119180928/t0(0) o250->MGC192.168.204.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2038.071849] LustreError: 66522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2038.084695] LustreError: 66522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2038.166672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2038.259890] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2042.427403] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2043.374520] LustreError: 66523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2048.481758] LustreError: 66522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2050.611115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2050.921696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2052.979345] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2054.962347] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2063.293325] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2068.706408] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2076.641376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2076.654649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2076.675082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2078.689379] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2078.692104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2078.708765] Lustre: Skipped 1 previous similar message [ 2078.709411] Lustre: Skipped 1 previous similar message [ 2082.616460] Lustre: server umount lustre-MDT0000 complete [ 2083.809303] LustreError: 68441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.818351] LustreError: 68441:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2086.617131] LustreError: 64666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153919 with bad export cookie 7377228225567401224 [ 2086.621928] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2086.625218] LustreError: 64666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2086.967154] Lustre: server umount lustre-MDT0001 complete [ 2093.785769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2103.664258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2104.288813] Lustre: 63596:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788153921/real 1788153921] req@000000001f3828fe x1875015119207168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788153937 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2104.328501] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.127172] Lustre: 63595:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788153924/real 1788153924] req@000000008e0a2e08 x1875015119207296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788153940 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2109.407386] Lustre: 63598:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788153926/real 1788153926] req@00000000f32c6788 x1875015119207424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788153942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2111.825289] LustreError: 69173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2111.910455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2111.973034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2112.927156] Lustre: 63598:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788153929/real 1788153929] req@000000004a83d4a9 x1875015119207680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788153945 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2116.161229] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2119.072137] Lustre: 63596:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788153935/real 1788153935] req@00000000dfe1076f x1875015119207808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788153951 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2120.160433] LustreError: 71123:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.174794] LustreError: 71123:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2124.360159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.302144] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2129.898303] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2129.907183] Lustre: Skipped 1 previous similar message [ 2134.053866] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2135.105273] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2141.223869] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2149.345849] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.357455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.365587] Lustre: Skipped 1 previous similar message [ 2149.375865] Lustre: Skipped 3 previous similar messages [ 2154.471146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2154.475911] Lustre: Skipped 2 previous similar messages [ 2163.167274] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2163.510648] Lustre: server umount lustre-MDT0000 complete [ 2164.737425] LustreError: 71122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2164.786830] LustreError: 71122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2167.812238] LustreError: 65408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154000 with bad export cookie 7377228225567402603 [ 2167.823271] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2167.827637] LustreError: 65408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2168.121942] Lustre: server umount lustre-MDT0001 complete [ 2175.649395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2179.666618] Lustre: DEBUG MARKER: files compared the same [ 2192.052600] Lustre: server umount lustre-OST0000 complete [ 2210.115811] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 2212.998967] Key type lgssc unregistered [ 2213.353372] LNet: 74398:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2213.363443] LNetError: 74398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2213.395021] LNet: Removed LNI 192.168.204.125@tcp [ 2214.245170] Key type .llcrypt unregistered [ 2214.249258] Key type ._llcrypt unregistered [ 2243.214077] Key type ._llcrypt registered [ 2243.216940] Key type .llcrypt registered [ 2243.399882] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2244.336828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2244.367075] alg: No test for adler32 (adler32-zlib) [ 2245.565397] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2245.865561] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 2247.679175] Key type lgssc registered [ 2249.379650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2257.991855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2258.024051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.455041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.056891] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug +malloc all [ 2269.665253] LustreError: 75813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.694218] LustreError: 75813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2272.603500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2273.019855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2277.510736] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug +malloc all [ 2285.912263] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2291.951380] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2299.155179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2299.369300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2304.494039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2305.286728] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug +malloc all [ 2312.766356] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2328.032133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2328.035302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2328.051982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2328.057727] Lustre: Skipped 1 previous similar message [ 2331.773636] Lustre: server umount lustre-OST0000 complete [ 2338.273824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2338.274899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2338.288615] Lustre: Skipped 1 previous similar message [ 2338.298324] Lustre: Skipped 1 previous similar message [ 2342.261895] Lustre: server umount lustre-MDT0000 complete [ 2343.403983] LustreError: 76543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.424197] LustreError: 76543:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2345.663580] LustreError: 75793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154178 with bad export cookie 15815958487275633508 [ 2345.672841] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2346.012748] Lustre: server umount lustre-MDT0001 complete [ 2358.333498] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 2361.694361] Key type lgssc unregistered [ 2362.171022] LNet: 79377:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2362.207510] LNetError: 79377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2362.235914] LNet: Removed LNI 192.168.204.125@tcp [ 2363.274799] Key type .llcrypt unregistered [ 2363.277765] Key type ._llcrypt unregistered [ 2390.021833] Key type ._llcrypt registered [ 2390.032531] Key type .llcrypt registered [ 2390.161581] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2391.211373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2391.231061] alg: No test for adler32 (adler32-zlib) [ 2392.268198] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2392.475141] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 2394.119171] Key type lgssc registered [ 2394.952464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2402.480949] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2402.512261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2418.079918] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2423.200279] LustreError: 80771:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.125@tcp: failed processing log, type 4: rc = -110 [ 2449.887237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2456.851764] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2466.648863] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2476.259489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2476.485204] Lustre: *** cfs_fail_loc=706, val=0*** [ 2476.951447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2477.031575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2482.149643] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2482.151570] LustreError: 80797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.175125] LustreError: 80797:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2485.226553] LustreError: 81855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.247128] LustreError: 81855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2490.344159] LustreError: 80797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2490.368103] LustreError: 80797:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2491.130455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.428419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2496.413890] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2504.254573] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2510.986705] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2516.965381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.965670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2516.991748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2517.033303] Lustre: Skipped 1 previous similar message [ 2522.090454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2522.100626] Lustre: Skipped 1 previous similar message [ 2527.218769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2527.234457] Lustre: Skipped 1 previous similar message [ 2528.223295] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2528.374165] Lustre: server umount lustre-OST0000 complete [ 2537.440863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2537.479570] Lustre: Skipped 1 previous similar message [ 2537.501436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2537.508539] Lustre: Skipped 1 previous similar message [ 2538.766367] Lustre: server umount lustre-MDT0000 complete [ 2542.562330] LustreError: 81856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.585106] LustreError: 81856:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2543.217408] LustreError: 80777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154376 with bad export cookie 13577753603013999204 [ 2543.231547] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2543.698757] Lustre: server umount lustre-MDT0001 complete [ 2556.256718] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 2558.949587] Key type lgssc unregistered [ 2559.230941] LNet: 84389:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2559.235706] LNetError: 84389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2559.248253] LNet: Removed LNI 192.168.204.125@tcp [ 2560.248811] Key type .llcrypt unregistered [ 2560.251543] Key type ._llcrypt unregistered [ 2590.423702] Key type ._llcrypt registered [ 2590.429325] Key type .llcrypt registered [ 2590.548737] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2592.271634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2592.304474] alg: No test for adler32 (adler32-zlib) [ 2593.518862] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2593.888125] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 2595.631195] Key type lgssc registered [ 2596.776679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2604.341852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2604.353792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2610.043868] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2618.072530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2618.620396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2623.918657] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2623.969379] LustreError: 86496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2623.999071] LustreError: 86496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2629.092633] LustreError: 86497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.122919] LustreError: 86497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2634.211066] LustreError: 86496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.225995] LustreError: 86496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2634.384354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2634.855280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2640.362222] LustreError: 86502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.376915] LustreError: 86502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2642.050310] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2648.552721] LustreError: 86501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.571280] LustreError: 86501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2652.659704] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2652.678719] Lustre: MGS: Client 5dfa8867-539c-4564-9cc4-78ae077d5353 (at 0@lo) reconnecting [ 2652.717606] LustreError: 86512:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2653.053191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2653.138652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2658.316528] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2675.169616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2675.186702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2675.207651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2678.755852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2678.756494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2678.756601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2678.786626] Lustre: Skipped 1 previous similar message [ 2680.297042] Lustre: server umount lustre-OST0000 complete [ 2688.993140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2688.993729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2689.013543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2689.021788] Lustre: Skipped 1 previous similar message [ 2689.045250] Lustre: Skipped 1 previous similar message [ 2694.113788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2694.123077] Lustre: Skipped 1 previous similar message [ 2699.234368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.236967] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 2699.249167] Lustre: Skipped 1 previous similar message [ 2699.347714] Lustre: server umount lustre-MDT0000 complete [ 2703.433443] Lustre: server umount lustre-MDT0001 complete [ 2707.574655] Lustre: server umount MGS complete [ 2719.667574] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:35:49 (1788154549) [ 2752.960061] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 2763.897257] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2799.995542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2803.834127] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2813.488115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2813.798925] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2813.871070] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2813.978178] Lustre: lustre-MDT0001: new disk, initializing [ 2814.054863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2814.085701] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2814.098457] Lustre: 93689:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2814.107267] LustreError: 93689:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2818.739699] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2830.852237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2831.222241] Lustre: lustre-OST0000: new disk, initializing [ 2831.232705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2831.240625] Lustre: 94454:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2831.335798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2838.372550] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2839.076559] Lustre: 94768:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000ef4a3cbb x1875015815441664/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788154687 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2848.857706] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2856.274244] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2856.291943] Lustre: MGS: Client df277526-9feb-4bfc-8ec1-100eb6d74f54 (at 0@lo) reconnecting [ 2856.361917] Lustre: 92957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2856.387302] LustreError: 93674:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2856.421858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2856.496292] Lustre: lustre-MDT0000: new disk, initializing [ 2856.597277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2856.629912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2856.644860] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2857.723061] Lustre: 95523:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000ce60574f x1875015815448448/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788154706 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2861.554443] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2862.888161] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2866.666747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2866.681201] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2866.937986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2892.264517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2892.265182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2892.276949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2892.278228] Lustre: Skipped 1 previous similar message [ 2898.479116] Lustre: server umount lustre-OST0000 complete [ 2907.617487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2907.618086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2908.816830] Lustre: server umount lustre-MDT0000 complete [ 2918.967812] Lustre: server umount lustre-MDT0001 complete [ 2922.405336] Lustre: server umount MGS complete [ 2933.113504] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:39:23 (1788154763) [ 2949.428851] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 2952.362301] Key type lgssc unregistered [ 2952.643221] LNet: 98034:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2952.652925] LNetError: 98034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2952.677736] LNet: Removed LNI 192.168.204.125@tcp [ 2953.525166] Key type .llcrypt unregistered [ 2953.529100] Key type ._llcrypt unregistered [ 2971.022595] Key type ._llcrypt registered [ 2971.024252] Key type .llcrypt registered [ 2971.120912] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2971.974936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2972.065665] alg: No test for adler32 (adler32-zlib) [ 2973.060294] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2973.239112] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 2974.919244] Key type lgssc registered [ 2975.921256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2982.719784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2982.767650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2984.204103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2988.550167] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2993.632908] LustreError: 99255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2993.650109] LustreError: 99255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2997.259954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2997.618528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3002.655381] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3010.747739] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3017.309794] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3024.051405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3024.484282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3028.604765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3030.572224] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3038.485780] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3049.440451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3049.454064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3049.467186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3053.538469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3053.540085] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3053.555792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3058.663961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3058.680202] Lustre: Skipped 2 previous similar messages [ 3063.263262] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3063.414429] Lustre: server umount lustre-OST0000 complete [ 3068.901954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3068.910661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3068.915737] Lustre: Skipped 1 previous similar message [ 3068.933729] Lustre: Skipped 1 previous similar message [ 3073.914486] Lustre: server umount lustre-MDT0000 complete [ 3074.023285] LustreError: 99251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3074.044659] LustreError: 99251:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3077.382954] LustreError: 99236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154910 with bad export cookie 1189071031907134907 [ 3077.384870] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3077.393845] LustreError: 99236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3077.705721] Lustre: server umount lustre-MDT0001 complete [ 3089.868304] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3093.333897] Key type lgssc unregistered [ 3093.656345] LNet: 102790:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3093.678516] LNetError: 102790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3093.707445] LNet: Removed LNI 192.168.204.125@tcp [ 3094.724420] Key type .llcrypt unregistered [ 3094.731087] Key type ._llcrypt unregistered [ 3112.546783] Key type ._llcrypt registered [ 3112.550888] Key type .llcrypt registered [ 3112.713763] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3114.125574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3114.203817] alg: No test for adler32 (adler32-zlib) [ 3115.256824] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3115.530177] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3117.311192] Key type lgssc registered [ 3118.451047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3125.298589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3125.303259] LustreError: 103873:0:(tgt_mount.c:2644:server_fill_super()) cfs_race id 716 sleeping [ 3125.304582] Lustre: Skipped 1 previous similar message [ 3125.316036] LustreError: 103874:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 waking [ 3125.320283] LustreError: 103873:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 3125.325620] LustreError: 103873:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3125.334685] LustreError: 103873:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3125.346660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3126.883651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3136.483183] LustreError: 103911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3136.501726] LustreError: 103911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3136.582065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3136.857221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3141.686295] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3146.539070] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lsmod [ 3149.138746] LustreError: 105074:0:(tgt_mount.c:2644:server_fill_super()) cfs_race id 716 sleeping [ 3149.148670] LustreError: 105080:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 waking [ 3149.153225] LustreError: 105074:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 3149.159379] LustreError: 105074:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3149.165245] LustreError: 105074:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3149.175811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3149.533290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3151.598506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3152.352704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3152.356918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3152.364776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3156.970314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3156.972396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.975321] Lustre: Skipped 1 previous similar message [ 3157.013776] Lustre: Skipped 1 previous similar message [ 3162.081975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3162.092899] Lustre: Skipped 3 previous similar messages [ 3166.175137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3166.406805] Lustre: server umount lustre-MDT0000 complete [ 3168.233942] LustreError: 103911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3168.264387] LustreError: 103911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3170.385834] LustreError: 103889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155003 with bad export cookie 10417013854480836424 [ 3170.401108] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3170.401529] LustreError: 103889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3171.359250] LustreError: 104343:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000514f962d x1875016362776320/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3171.567655] Lustre: server umount lustre-MDT0001 complete [ 3186.666690] Lustre: server umount lustre-OST0000 complete [ 3196.268531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3196.872212] LustreError: 106148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3197.007430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3202.033264] LustreError: 106149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3202.176705] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3207.137842] LustreError: 106148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3212.138366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3212.631512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3217.236865] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3225.586662] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3232.161728] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3239.975790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3240.368975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3241.403241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3247.185343] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3256.511279] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3267.042838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3267.055087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3267.078964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3267.084910] Lustre: Skipped 3 previous similar messages [ 3268.078215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3271.367811] Lustre: server umount lustre-OST0000 complete [ 3278.310994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3278.327419] Lustre: Skipped 1 previous similar message [ 3278.334833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3278.343838] Lustre: Skipped 3 previous similar messages [ 3281.791494] Lustre: server umount lustre-MDT0000 complete [ 3283.428919] LustreError: 106149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.466138] LustreError: 106149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3286.021099] LustreError: 106129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155118 with bad export cookie 10417013854480837355 [ 3286.035941] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3286.042157] LustreError: 106129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3286.470337] Lustre: server umount lustre-MDT0001 complete [ 3299.430439] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3302.516684] Key type lgssc unregistered [ 3302.895920] LNet: 109683:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3302.904168] LNetError: 109683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3302.934967] LNet: Removed LNI 192.168.204.125@tcp [ 3303.767944] Key type .llcrypt unregistered [ 3303.771713] Key type ._llcrypt unregistered [ 3328.878654] Key type ._llcrypt registered [ 3328.880390] Key type .llcrypt registered [ 3329.025163] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3330.168509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3330.239875] alg: No test for adler32 (adler32-zlib) [ 3331.299839] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3331.612711] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3333.287282] Key type lgssc registered [ 3334.484314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3342.310067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3342.323670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3343.810855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3348.042614] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3354.091211] LustreError: 111101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.114277] LustreError: 111101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3357.099900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.613706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3363.068318] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3371.474709] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3377.285891] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3384.341244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3384.650993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.801468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3391.187827] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3399.300358] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3408.001354] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3412.233572] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3418.597875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3418.600538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3418.611213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3418.612928] Lustre: Skipped 1 previous similar message [ 3418.623128] Lustre: Skipped 1 previous similar message [ 3423.714656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3423.718240] Lustre: Skipped 1 previous similar message [ 3428.834306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3428.838032] Lustre: Skipped 1 previous similar message [ 3430.367175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3430.546237] Lustre: server umount lustre-OST0000 complete [ 3434.465763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3434.481425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3434.508115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3439.073069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3439.074596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3442.634497] Lustre: server umount lustre-MDT0000 complete [ 3444.192356] LustreError: 113565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3444.203898] LustreError: 113565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3445.940954] LustreError: 111084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155278 with bad export cookie 17052796455350399211 [ 3445.948577] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3445.950672] LustreError: 111084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3446.242938] Lustre: server umount lustre-MDT0001 complete [ 3457.167974] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3459.766820] Key type lgssc unregistered [ 3460.014330] LNet: 114743:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3460.023298] LNetError: 114743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3460.034912] LNet: Removed LNI 192.168.204.125@tcp [ 3460.832952] Key type .llcrypt unregistered [ 3460.836429] Key type ._llcrypt unregistered [ 3478.460967] Key type ._llcrypt registered [ 3478.464111] Key type .llcrypt registered [ 3478.570516] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3479.292249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3479.380098] alg: No test for adler32 (adler32-zlib) [ 3480.360156] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3480.514220] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3482.223876] Key type lgssc registered [ 3483.732667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3492.267492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3492.297822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3493.742147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3498.251886] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3504.102813] LustreError: 115967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3504.151923] LustreError: 115967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3509.222029] LustreError: 115968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3509.354308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3509.716942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3513.690185] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3523.114784] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3528.841270] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3535.440446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3535.831111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3538.924708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3541.661294] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3550.607547] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3564.519879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3564.522603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3564.528583] LustreError: Skipped 1 previous similar message [ 3564.543502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3569.637912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3569.648079] Lustre: Skipped 2 previous similar messages [ 3570.578582] Lustre: server umount lustre-OST0000 complete [ 3574.774356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3574.790460] Lustre: Skipped 1 previous similar message [ 3574.801696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3574.810329] Lustre: Skipped 1 previous similar message [ 3579.876099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3579.889234] Lustre: Skipped 1 previous similar message [ 3580.134280] Lustre: server umount lustre-MDT0000 complete [ 3583.762066] LustreError: 115948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155416 with bad export cookie 14890104008011240805 [ 3583.769713] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3583.775721] LustreError: 115948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3584.061107] Lustre: server umount lustre-MDT0001 complete [ 3595.413739] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3597.914468] Key type lgssc unregistered [ 3598.174108] LNet: 119525:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3598.180396] LNetError: 119525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3598.199864] LNet: Removed LNI 192.168.204.125@tcp [ 3599.121225] Key type .llcrypt unregistered [ 3599.123102] Key type ._llcrypt unregistered [ 3620.791350] Key type ._llcrypt registered [ 3620.795416] Key type .llcrypt registered [ 3620.927236] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3621.554153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3621.579399] alg: No test for adler32 (adler32-zlib) [ 3622.632431] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3622.888197] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3624.599173] Key type lgssc registered [ 3625.885335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3634.137658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3634.160682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3635.631407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3640.161772] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3645.924603] LustreError: 120936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3645.936524] LustreError: 120936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3647.951860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3648.260517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3652.770512] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3660.273545] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3666.276771] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3673.420281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3673.796989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3679.890695] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3681.016788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3686.943334] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3696.917809] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3699.903885] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3699.907745] Lustre: Skipped 2 previous similar messages [ 3703.964468] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3704.007264] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3704.301908] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3704.311552] Lustre: Skipped 2 previous similar messages [ 3710.508905] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3710.515888] Lustre: Skipped 1 previous similar message [ 3716.508282] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.25@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3716.524641] Lustre: Skipped 2 previous similar messages [ 3718.597781] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.25@tcp 192.168.0.[2,10]@tcp [ 3729.491215] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.25/24@tcp in log lustre-MDT0000 [ 3729.501416] Lustre: Skipped 2 previous similar messages [ 3738.689095] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.25/24@tcp [ 3738.695353] Lustre: Skipped 1 previous similar message [ 3747.598888] Lustre: server umount lustre-OST0000 complete [ 3753.956229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3753.971711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3753.979559] Lustre: Skipped 1 previous similar message [ 3759.073514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3759.081337] Lustre: Skipped 1 previous similar message [ 3762.237131] Lustre: server umount lustre-MDT0000 complete [ 3764.191110] Lustre: 120246:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155581/real 1788155581] req@00000000ea516521 x1875016894450816/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788155597 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3764.192358] LustreError: 120936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3764.230626] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3764.261232] LustreError: 120936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3764.313851] Lustre: Skipped 1 previous similar message [ 3765.864043] LustreError: 120918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155598 with bad export cookie 2252298351320110688 [ 3765.864043] LustreError: 120916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155598 with bad export cookie 2252298351320110688 [ 3765.864780] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3765.885966] LustreError: 120916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3766.258957] Lustre: server umount lustre-MDT0001 complete [ 3776.266251] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3779.220733] Key type lgssc unregistered [ 3779.495763] LNet: 125977:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3779.499679] LNetError: 125977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3779.523557] LNet: Removed LNI 192.168.204.125@tcp [ 3780.433904] Key type .llcrypt unregistered [ 3780.437165] Key type ._llcrypt unregistered [ 3806.568089] Key type ._llcrypt registered [ 3806.571846] Key type .llcrypt registered [ 3806.680815] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3807.932398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3807.955516] alg: No test for adler32 (adler32-zlib) [ 3808.985328] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3809.297267] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3810.991211] Key type lgssc registered [ 3812.044702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3820.242095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3820.257481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3821.741907] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3821.788759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3821.801893] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.25/24@tcp [ 3825.618342] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3831.268455] LustreError: 127539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3831.298229] LustreError: 127539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3834.941357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3835.561867] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3835.630379] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.25/24@tcp [ 3835.680152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3841.654857] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3850.035697] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3856.195786] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3864.202341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3864.536875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3866.627215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 3870.838680] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3877.108557] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3896.288295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3896.309450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3896.322672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3896.333058] Lustre: Skipped 1 previous similar message [ 3896.355233] Lustre: Skipped 1 previous similar message [ 3899.411407] Lustre: server umount lustre-OST0000 complete [ 3906.535282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3906.543948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3906.555895] Lustre: Skipped 1 previous similar message [ 3906.564832] Lustre: Skipped 1 previous similar message [ 3909.360254] Lustre: server umount lustre-MDT0000 complete [ 3911.658952] LustreError: 127539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3911.681952] LustreError: 127539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3913.257217] LustreError: 127519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155746 with bad export cookie 3872380657909278889 [ 3913.259426] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3913.279067] LustreError: 127519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3913.725908] Lustre: server umount lustre-MDT0001 complete [ 3926.686888] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3930.456883] Key type lgssc unregistered [ 3930.757685] LNet: 131127:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3930.766035] LNetError: 131127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3930.782916] LNet: Removed LNI 192.168.204.125@tcp [ 3931.795473] Key type .llcrypt unregistered [ 3931.798093] Key type ._llcrypt unregistered [ 3959.465966] Key type ._llcrypt registered [ 3959.468507] Key type .llcrypt registered [ 3959.549262] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 3974.445168] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3975.639116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3975.707143] alg: No test for adler32 (adler32-zlib) [ 3976.705781] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3976.935760] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3978.631198] Key type lgssc registered [ 3979.906883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4015.385931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4015.415756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4016.652739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4016.672112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4016.761706] Lustre: lustre-MDT0000: new disk, initializing [ 4016.843981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4016.862614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4021.757194] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4032.813844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4032.951465] Lustre: 134530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4033.045162] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4033.068131] Lustre: Skipped 1 previous similar message [ 4033.253061] Lustre: lustre-MDT0001: new disk, initializing [ 4033.435460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4033.511882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4033.521119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4037.947754] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4048.207696] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4053.964282] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4060.924812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4061.185346] Lustre: lustre-OST0000: new disk, initializing [ 4061.190315] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4061.197549] Lustre: 136499:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4061.280920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.037671] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4071.474586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4071.486416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4071.616791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4077.736284] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4082.290759] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4082.500550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4086.930444] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4087.155051] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4092.388946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4092.393495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4092.401294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4092.404681] Lustre: Skipped 1 previous similar message [ 4092.423517] LustreError: Skipped 1 previous similar message [ 4095.616274] Lustre: server umount lustre-OST0000 complete [ 4100.064830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4100.073893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4100.092733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4100.097028] Lustre: Skipped 1 previous similar message [ 4102.627477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4102.627799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4106.010740] Lustre: server umount lustre-MDT0000 complete [ 4110.403759] LustreError: 135325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155943 with bad export cookie 12871347476860530140 [ 4110.406514] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4110.420140] LustreError: 135325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4110.797895] Lustre: server umount lustre-MDT0001 complete [ 4120.290381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4120.774093] LustreError: 138715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4120.789111] LustreError: 138715:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4120.861190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4125.312887] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4126.180938] LustreError: 138716:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4130.276958] LustreError: 138715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4134.357304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4134.704257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4139.761838] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4148.128853] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4154.321763] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4161.952954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4162.355275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4168.801596] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4177.373981] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4181.929207] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4182.218919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4186.851897] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4187.128068] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4195.677939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4195.949575] Lustre: lustre-OST0001: new disk, initializing [ 4195.952742] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4195.959605] Lustre: 142086:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4196.050881] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4202.517299] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4205.189647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4205.199684] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4205.314124] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4211.446497] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4211.677332] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4216.948180] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4217.149962] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4223.607384] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff900b910b0000.ost_server_uuid 50 [ 4225.175652] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff900b910b0000.ost_server_uuid in IDLE state after 0 sec [ 4236.258173] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4236.262124] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4236.271724] LustreError: Skipped 1 previous similar message [ 4236.284138] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4236.290479] Lustre: Skipped 1 previous similar message [ 4236.300323] Lustre: Skipped 2 previous similar messages [ 4236.827830] Lustre: server umount lustre-OST0001 complete [ 4241.380943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4241.382622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4241.387568] LustreError: 140541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4241.394440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4241.425216] Lustre: Skipped 1 previous similar message [ 4246.543078] LustreError: 142396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4246.559357] LustreError: 142396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4251.620125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4251.628483] Lustre: Skipped 4 previous similar messages [ 4255.199768] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4255.434507] Lustre: server umount lustre-OST0000 complete [ 4261.895904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4265.464828] Lustre: server umount lustre-MDT0000 complete [ 4266.976307] LustreError: 138715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4266.986620] LustreError: 138715:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4268.802591] LustreError: 138695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156101 with bad export cookie 12871347476860531022 [ 4268.807129] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4268.810387] LustreError: 138695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4269.029827] Lustre: server umount lustre-MDT0001 complete [ 4289.361981] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 4291.924988] Key type lgssc unregistered [ 4292.255942] LNet: 145104:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4292.263223] LNetError: 145104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4292.285160] LNet: Removed LNI 192.168.204.125@tcp [ 4293.282783] Key type .llcrypt unregistered [ 4293.292921] Key type ._llcrypt unregistered [ 4339.145909] Key type ._llcrypt registered [ 4339.154800] Key type .llcrypt registered [ 4339.328425] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 4355.093337] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 4356.229740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4356.392549] alg: No test for adler32 (adler32-zlib) [ 4357.508330] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4357.698777] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 4359.343243] Key type lgssc registered [ 4360.360794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4398.165060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4398.202366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4399.426935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4399.455508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4399.540557] Lustre: lustre-MDT0000: new disk, initializing [ 4399.598328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4399.611835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4404.750714] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4414.772399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4414.883777] Lustre: 149273:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4414.975976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4414.982845] Lustre: Skipped 1 previous similar message [ 4415.069474] Lustre: lustre-MDT0001: new disk, initializing [ 4415.166398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4415.192714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4415.199297] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4420.139957] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4429.754530] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4435.965737] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4442.785161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4443.046606] Lustre: lustre-OST0000: new disk, initializing [ 4443.053498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4443.060333] Lustre: 151241:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4443.170119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4449.184592] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4451.414090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4451.425647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4451.611706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4460.641228] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4466.171174] Lustre: Failing over lustre-OST0000 [ 4466.353994] Lustre: server umount lustre-OST0000 complete [ 4482.527160] Lustre: 147495:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156299/real 1788156299] req@00000000db5f0d3a x1875017665118848/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788156315 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4482.527452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4482.560519] Lustre: 147495:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4484.576812] Lustre: 147492:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156301/real 1788156301] req@00000000a0a1feec x1875017665119616/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788156317 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4484.657380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4485.011893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4485.041761] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4485.220118] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4489.055503] Lustre: 147493:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156305/real 1788156305] req@00000000783e5ea7 x1875017665120640/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788156321 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4489.075135] Lustre: 147493:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4491.216510] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4493.239484] Lustre: lustre-OST0000: Recovery over after 0:08, of 3 clients 3 recovered and 0 were evicted. [ 4493.239988] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4494.303215] Lustre: 147494:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156310/real 1788156310] req@00000000a468bef6 x1875017665121792/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788156326 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4494.346114] Lustre: 147494:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4495.041709] Lustre: Failing over lustre-MDT0000 [ 4495.232805] Lustre: server umount lustre-MDT0000 complete [ 4497.378061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4497.381941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4497.392718] Lustre: Skipped 1 previous similar message [ 4498.406105] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4498.423092] Lustre: Skipped 1 previous similar message [ 4499.487165] Lustre: 147494:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156315/real 1788156315] req@00000000da79ce8a x1875017665122816/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788156331 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4500.536431] LustreError: 150689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4500.556210] LustreError: 150689:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4503.523758] LustreError: 149282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4503.540254] LustreError: 149282:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4505.651145] LustreError: 149278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4508.651131] LustreError: 149283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4508.669944] LustreError: 149283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4513.762041] LustreError: 149282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4513.783448] LustreError: 149282:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4514.015644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4514.143319] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4514.371978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4514.441427] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4515.890678] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4518.616275] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4519.406860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4519.417950] Lustre: Skipped 1 previous similar message [ 4519.439749] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4519.516652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4529.120625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4529.131838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4529.142481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4529.650885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4534.064513] Lustre: server umount lustre-OST0000 complete [ 4539.875863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4539.879378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4539.894991] Lustre: Skipped 1 previous similar message [ 4539.896934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4544.553788] Lustre: server umount lustre-MDT0000 complete [ 4544.993266] LustreError: 149278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4545.030917] LustreError: 149278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4549.294820] LustreError: 149264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156382 with bad export cookie 8834150982097883302 [ 4549.297501] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4549.307803] LustreError: 149264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4549.629573] Lustre: server umount lustre-MDT0001 complete [ 4562.432825] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 4565.744062] Key type lgssc unregistered [ 4566.125107] LNet: 154893:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4566.134909] LNetError: 154893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4566.151212] LNet: Removed LNI 192.168.204.125@tcp [ 4567.150521] Key type .llcrypt unregistered [ 4567.152424] Key type ._llcrypt unregistered [ 4605.632489] Key type ._llcrypt registered [ 4605.637621] Key type .llcrypt registered [ 4605.795224] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 4606.920076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4607.046211] alg: No test for adler32 (adler32-zlib) [ 4608.205377] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4608.476310] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 4610.199269] Key type lgssc registered [ 4611.322926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4619.693842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4619.711718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4621.186855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4625.915557] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4630.506356] LustreError: 156415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4630.531647] LustreError: 156415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4634.288907] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4635.625322] LustreError: 156416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4637.746221] LustreError: 156445:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4637.754553] LustreError: 156445:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4637.761835] LustreError: 156445:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4638.173986] Lustre: server umount lustre-MDT0000 complete [ 4651.053082] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 4661.501841] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 4699.472958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4699.802831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4699.865108] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4699.978857] Lustre: lustre-MDT0000: new disk, initializing [ 4700.086901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4700.105107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4705.294790] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4717.822116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4717.990993] Lustre: 159749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4718.060882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4718.070174] Lustre: Skipped 1 previous similar message [ 4718.228538] Lustre: lustre-MDT0001: new disk, initializing [ 4718.399495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4718.467871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4718.489160] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4724.646395] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4736.796353] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4743.118920] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4749.762435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4750.092473] Lustre: lustre-OST0000: new disk, initializing [ 4750.097436] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4750.104559] Lustre: 161719:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4750.198457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4756.544578] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4759.633267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4759.648716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4759.771081] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4767.239922] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4772.176556] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4774.123942] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:10:04 (1788156604) [ 5062.112724] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5062.122580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5062.124884] Lustre: Skipped 1 previous similar message [ 5067.233882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5067.244261] Lustre: Skipped 4 previous similar messages [ 5068.350971] Lustre: server umount lustre-MDT0000 complete [ 5072.353042] LustreError: 159759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5072.369936] LustreError: 159759:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5072.572141] LustreError: 159740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156905 with bad export cookie 11966367044692409874 [ 5072.582123] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5072.585954] LustreError: 159740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5077.477498] LustreError: 162184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5077.477611] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5077.503606] Lustre: Skipped 1 previous similar message [ 5077.509062] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5079.014086] Lustre: server umount lustre-MDT0001 complete [ 5090.023435] Lustre: server umount lustre-OST0000 complete [ 5101.102962] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 5113.161579] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5158.784324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5159.223315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5159.317609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5159.400194] Lustre: lustre-MDT0000: new disk, initializing [ 5159.578687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5159.613164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5165.538083] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5177.559960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5177.715708] Lustre: 166397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5177.780593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5177.784518] Lustre: Skipped 1 previous similar message [ 5177.909308] Lustre: lustre-MDT0001: new disk, initializing [ 5177.991181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5178.034919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5178.048886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5183.750486] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5194.759798] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5201.335523] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5208.835539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5209.217430] Lustre: lustre-OST0000: new disk, initializing [ 5209.220890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5209.227173] Lustre: 168361:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5209.342806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5216.309375] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5218.396523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5218.408394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5218.514871] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5226.025844] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5230.924260] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5231.185955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5236.630136] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5236.927561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5239.267690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5239.271312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5239.278380] LustreError: Skipped 1 previous similar message [ 5239.295477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5239.307356] Lustre: Skipped 1 previous similar message [ 5244.405357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5244.414711] Lustre: Skipped 1 previous similar message [ 5245.453499] Lustre: server umount lustre-OST0000 complete [ 5254.632082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5254.635419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5254.643564] Lustre: Skipped 2 previous similar messages [ 5256.482654] Lustre: server umount lustre-MDT0000 complete [ 5259.744542] LustreError: 166406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5259.783925] LustreError: 166406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5260.924197] LustreError: 166387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157093 with bad export cookie 11966367044692568704 [ 5260.935125] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5261.366412] Lustre: server umount lustre-MDT0001 complete [ 5273.415442] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:18:22 (1788157102) [ 5291.324688] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 5301.824412] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5347.067287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5347.567195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5347.673215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5347.915189] Lustre: lustre-MDT0000: new disk, initializing [ 5348.036130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5348.094421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5354.735113] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5366.991290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5367.261087] Lustre: 173123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5367.667703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5372.502616] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5385.965058] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5393.296959] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5402.394096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5402.835183] Lustre: lustre-OST0000: new disk, initializing [ 5402.841956] Lustre: Skipped 1 previous similar message [ 5402.851797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5402.861697] Lustre: Skipped 2 previous similar messages [ 5402.869705] Lustre: 175101:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5410.229296] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5410.398138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5410.408229] Lustre: Skipped 1 previous similar message [ 5410.421784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5410.612317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5422.129386] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5436.388927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5436.396228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5436.398061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5436.398068] Lustre: Skipped 1 previous similar message [ 5436.448828] Lustre: Skipped 1 previous similar message [ 5438.119768] Lustre: server umount lustre-OST0000 complete [ 5444.587679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5444.610398] LustreError: Skipped 1 previous similar message [ 5444.627073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.193434] Lustre: server umount lustre-MDT0000 complete [ 5451.749571] LustreError: 174497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5451.788822] LustreError: 174497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5455.432762] LustreError: 173115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157288 with bad export cookie 11966367044692569586 [ 5455.435580] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5455.448772] LustreError: 173115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5455.972738] Lustre: server umount lustre-MDT0001 complete [ 5467.802955] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:21:37 (1788157297) [ 5489.322926] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 5501.637242] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5541.558385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.759271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5541.785418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5541.901197] Lustre: lustre-MDT0000: new disk, initializing [ 5542.030374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5542.038692] Lustre: Skipped 2 previous similar messages [ 5542.051989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5547.514532] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5558.384285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5558.549826] Lustre: 179430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5558.895193] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5563.023765] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5572.365476] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5578.219533] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5586.214063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5586.636348] Lustre: 181406:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5594.207587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5594.338815] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5594.467280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5605.968993] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5615.074741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5615.083119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5615.105825] Lustre: Skipped 1 previous similar message [ 5615.115186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5615.131443] Lustre: Skipped 5 previous similar messages [ 5628.384430] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5628.682891] Lustre: server umount lustre-OST0000 complete [ 5635.552808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5635.560534] LustreError: Skipped 1 previous similar message [ 5647.329455] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5647.694748] Lustre: server umount lustre-MDT0000 complete [ 5650.912555] LustreError: 179437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5650.927553] LustreError: 179437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5652.656214] LustreError: 179422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157485 with bad export cookie 11966367044692570671 [ 5652.672866] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5653.322412] Lustre: server umount lustre-MDT0001 complete [ 5666.713549] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 5670.021648] Key type lgssc unregistered [ 5670.277708] LNet: 183266:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5670.284977] LNetError: 183266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5670.307200] LNet: Removed LNI 192.168.204.125@tcp [ 5671.313611] Key type .llcrypt unregistered [ 5671.328584] Key type ._llcrypt unregistered [ 5698.077617] Key type ._llcrypt registered [ 5698.079873] Key type .llcrypt registered [ 5698.194835] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5699.416566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5699.459377] alg: No test for adler32 (adler32-zlib) [ 5700.611863] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5700.865708] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 5702.567231] Key type lgssc registered [ 5703.788160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5711.859657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5711.880504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5713.371959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5718.638754] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5723.624332] LustreError: 184682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5723.650166] LustreError: 184682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5728.196971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5728.707891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5734.316989] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5743.742689] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5750.345245] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5758.250925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5758.454839] LustreError: 186505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5758.727376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5764.093069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5765.946769] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5774.315965] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5793.769018] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5793.785447] Lustre: Skipped 1 previous similar message [ 5793.788688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5793.803764] Lustre: Skipped 1 previous similar message [ 5798.906725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5798.911737] Lustre: Skipped 1 previous similar message [ 5799.576814] Lustre: server umount lustre-OST0000 complete [ 5805.548890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5805.558076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5805.575778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5807.008360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5808.032866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5808.040396] Lustre: Skipped 3 previous similar messages [ 5810.611473] Lustre: server umount lustre-MDT0000 complete [ 5813.217083] LustreError: 184682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5813.246968] LustreError: 184682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5814.660302] LustreError: 184662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157647 with bad export cookie 3482743827618792796 [ 5814.677759] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5814.683029] LustreError: 184662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5815.129996] Lustre: server umount lustre-MDT0001 complete [ 5828.624898] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 5831.472640] Key type lgssc unregistered [ 5831.793464] LNet: 188240:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5831.809339] LNetError: 188240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5831.832880] LNet: Removed LNI 192.168.204.125@tcp [ 5832.744511] Key type .llcrypt unregistered [ 5832.748623] Key type ._llcrypt unregistered [ 5862.173216] Key type ._llcrypt registered [ 5862.174888] Key type .llcrypt registered [ 5862.318675] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5863.458645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5863.494072] alg: No test for adler32 (adler32-zlib) [ 5864.587982] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5864.934900] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 5866.640070] Key type lgssc registered [ 5867.709552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5875.840593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5875.885342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5877.340163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5882.198450] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5887.458123] LustreError: 189657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5887.491708] LustreError: 189657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5891.664306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5892.053157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5896.963783] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5903.482506] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5909.548541] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5918.164277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5918.662237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5922.751627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5926.084644] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5933.792915] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5946.974076] Lustre: lustre-OST0000: Not available for connect from 192.168.204.25@tcp (stopping) [ 5947.366358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5947.379171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5947.400358] Lustre: Skipped 1 previous similar message [ 5950.314116] Lustre: server umount lustre-OST0000 complete [ 5959.720544] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff900b872f9800.ost_server_uuid 50 [ 5962.159566] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5978.081495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5978.107114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5978.111206] Lustre: Skipped 2 previous similar messages [ 5983.203826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5983.210473] Lustre: Skipped 1 previous similar message [ 5983.350925] Lustre: server umount lustre-MDT0000 complete [ 5987.303905] LustreError: 189652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5987.352297] LustreError: 189652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5988.332952] LustreError: 189639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157821 with bad export cookie 4218306407972765197 [ 5988.345937] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5988.351372] LustreError: 189639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5988.723639] Lustre: server umount lustre-MDT0001 complete [ 5998.018350] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:30:27 (1788157827) [ 6005.311278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6005.883459] LustreError: 193574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6005.963089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6011.127838] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6011.376887] LustreError: 193575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6015.461976] LustreError: 193574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6020.388319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6020.800907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6025.333861] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6032.964678] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6039.070916] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6045.852577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6046.147617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6050.255361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6053.008515] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6060.078135] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6068.950165] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6069.348293] Lustre: lustre-OST0001: new disk, initializing [ 6069.356658] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6069.379481] Lustre: 196417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6069.602893] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6076.351899] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6078.621461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6078.631490] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6078.754499] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6087.488681] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6091.232667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6091.237931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6091.247702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6091.249589] Lustre: Skipped 2 previous similar messages [ 6091.287965] Lustre: Skipped 1 previous similar message [ 6097.041908] Lustre: server umount lustre-OST0000 complete [ 6098.498333] LustreError: 195398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6098.524258] LustreError: 195398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6103.361335] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6103.556773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6107.299941] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6107.496383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6114.791913] LustreError: 195397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6114.815269] LustreError: 195397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6125.028831] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6125.029672] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6125.030675] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6125.030681] Lustre: Skipped 3 previous similar messages [ 6125.036045] LustreError: Skipped 2 previous similar messages [ 6125.042637] Lustre: Skipped 1 previous similar message [ 6127.574794] Lustre: server umount lustre-OST0001 complete [ 6133.729885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6133.742476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6133.764235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6133.769991] Lustre: Skipped 1 previous similar message [ 6137.269533] Lustre: server umount lustre-MDT0000 complete [ 6140.771456] LustreError: 193554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157973 with bad export cookie 4218306407972766268 [ 6140.773693] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6140.780387] LustreError: 193554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6141.169800] Lustre: server umount lustre-MDT0001 complete [ 6167.476505] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:33:17 (1788157997) [ 6174.870092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6175.062140] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6175.096691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6175.223035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6180.389957] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6188.583995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6188.682445] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6188.776513] Lustre: 199857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6194.013722] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6203.789268] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6210.106665] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6218.609783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6218.912167] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6226.655357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6226.871343] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6236.220990] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6244.601571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6244.672284] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6244.988249] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6244.996537] Lustre: Skipped 2 previous similar messages [ 6251.913151] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6260.056757] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6265.947173] Lustre: lustre-OST0000: Not available for connect from 192.168.204.25@tcp (stopping) [ 6265.959047] Lustre: Skipped 2 previous similar messages [ 6267.872227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6267.883813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6267.904300] Lustre: Skipped 2 previous similar messages [ 6269.418309] Lustre: server umount lustre-OST0000 complete [ 6271.030746] LustreError: 201711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6271.059472] LustreError: 201711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6288.352064] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6288.355270] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6288.371032] LustreError: Skipped 2 previous similar messages [ 6288.386477] Lustre: Skipped 1 previous similar message [ 6292.075971] Lustre: server umount lustre-OST0001 complete [ 6298.607768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6298.622748] Lustre: Skipped 5 previous similar messages [ 6302.482770] Lustre: server umount lustre-MDT0000 complete [ 6306.499198] LustreError: 199851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158139 with bad export cookie 4218306407972767675 [ 6306.501333] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6306.517985] LustreError: 199851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6306.924504] Lustre: server umount lustre-MDT0001 complete [ 6333.969927] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:36:03 (1788158163) [ 6351.825190] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 6365.202620] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 6406.665323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6406.809302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6406.840754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6406.947991] Lustre: lustre-MDT0000: new disk, initializing [ 6407.010463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6407.023675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6412.374857] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6426.124934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6426.291958] Lustre: 207959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6426.352779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6426.360111] Lustre: Skipped 1 previous similar message [ 6426.451582] Lustre: lustre-MDT0001: new disk, initializing [ 6426.644323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6426.667081] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6431.925111] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6442.096787] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6447.668980] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6455.972116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6456.270360] Lustre: lustre-OST0000: new disk, initializing [ 6456.273368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6456.280298] Lustre: 209929:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6462.852694] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6463.049259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6463.056506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6463.157451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6474.170660] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6481.484157] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6481.788765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6486.912214] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6487.194889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6494.177480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6494.178664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6494.190156] LustreError: Skipped 2 previous similar messages [ 6494.191438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6494.234552] Lustre: Skipped 3 previous similar messages [ 6496.084312] Lustre: server umount lustre-OST0000 complete [ 6506.591031] Lustre: server umount lustre-MDT0000 complete [ 6509.537910] LustreError: 210209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6509.569567] LustreError: 210209:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6510.599266] LustreError: 207952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158343 with bad export cookie 4218306407972769236 [ 6510.612050] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6510.614185] LustreError: 207952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6511.059855] Lustre: server umount lustre-MDT0001 complete [ 6520.394727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6526.300320] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6535.831483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6541.247429] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6549.791289] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6555.689540] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6562.962419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6569.323345] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6577.138221] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6582.181921] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6582.412675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6587.332761] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6587.539441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6590.944681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6590.957913] LustreError: Skipped 1 previous similar message [ 6590.965095] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6590.986744] Lustre: Skipped 2 previous similar messages [ 6595.844295] Lustre: server umount lustre-OST0000 complete [ 6603.460657] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6608.815113] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6637.516803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6642.028687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6642.060429] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6644.186890] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6651.203819] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6655.156540] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6656.363222] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6656.378190] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6661.600598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6661.620840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6661.623546] LustreError: Skipped 1 previous similar message [ 6661.660496] Lustre: Skipped 8 previous similar messages [ 6665.792150] Lustre: server umount lustre-OST0000 complete [ 6676.538486] Lustre: server umount lustre-MDT0000 complete [ 6676.969438] LustreError: 212141:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6676.983880] LustreError: 212141:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6680.015051] LustreError: 212126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158512 with bad export cookie 4218306407972770118 [ 6680.021166] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6680.024029] LustreError: 212126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6680.473520] Lustre: server umount lustre-MDT0001 complete [ 6690.015242] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:42:00 (1788158520) [ 6697.392196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6698.143439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6698.151243] Lustre: Skipped 6 previous similar messages [ 6703.579549] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6713.068544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6718.147278] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6726.451254] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6733.479275] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6740.997925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6747.833259] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6755.013951] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6760.109700] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6760.393023] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6767.944036] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6768.376858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6777.703923] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6777.912991] Lustre: lustre-OST0001: new disk, initializing [ 6777.916712] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6777.923305] Lustre: 221486:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6784.645816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6784.660331] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6784.817241] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6786.044485] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6796.191143] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6801.530080] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6801.664196] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6806.243178] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6806.460901] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6810.594188] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6810.597081] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6810.612271] LustreError: Skipped 1 previous similar message [ 6810.639361] Lustre: Skipped 6 previous similar messages [ 6814.458580] Lustre: server umount lustre-OST0001 complete [ 6821.398947] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6821.732835] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6826.184576] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6826.445164] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6856.256861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6861.807178] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6861.821419] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6861.837638] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6861.852653] Lustre: Skipped 1 previous similar message [ 6863.599970] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6871.897766] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6873.272159] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6881.388040] Lustre: server umount lustre-OST0001 complete [ 6904.620866] Lustre: server umount lustre-OST0000 complete [ 6918.113313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6918.121842] Lustre: Skipped 12 previous similar messages [ 6923.231596] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6923.598821] Lustre: server umount lustre-MDT0000 complete [ 6927.863189] LustreError: 218091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158760 with bad export cookie 4218306407972771084 [ 6927.865565] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6927.872877] LustreError: 218091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6942.176089] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6942.432494] Lustre: server umount lustre-MDT0001 complete [ 6967.255852] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:46:36 (1788158796) [ 6974.990684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6975.338380] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6975.379492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6980.441509] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6989.417155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6989.534040] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6989.560828] Lustre: 226727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6994.411307] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7001.667682] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7007.810404] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7014.944816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7015.086035] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7021.894628] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7031.196514] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7040.754859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7040.824139] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7047.982686] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7056.474647] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7057.943482] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7061.767935] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7061.964237] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7066.814105] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7067.029611] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7073.434313] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff900b83872000.ost_server_uuid 50 [ 7075.101573] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff900b83872000.ost_server_uuid in IDLE state after 0 sec [ 7076.560822] Lustre: Permanently deactivating lustre-OST0001 [ 7076.579383] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7079.650528] Lustre: Permanently reactivating lustre-OST0001 [ 7087.379369] Lustre: server umount lustre-OST0001 complete [ 7088.618693] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7088.623392] LustreError: 228574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7088.640910] Lustre: Skipped 7 previous similar messages [ 7088.651102] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7088.659245] LustreError: 228574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [ 7088.688747] LustreError: Skipped 5 previous similar messages [ 7097.132240] Lustre: server umount lustre-OST0000 complete [ 7107.428814] Lustre: server umount lustre-MDT0000 complete [ 7111.748608] LustreError: 226722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158944 with bad export cookie 4218306407972772372 [ 7111.754544] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7111.760185] LustreError: 226722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7112.098544] Lustre: server umount lustre-MDT0001 complete [ 7140.027670] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:49:29 (1788158969) [ 7148.625892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7149.064589] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7149.108693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7149.118202] Lustre: Skipped 5 previous similar messages [ 7154.586708] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7164.658863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7164.739767] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7164.813081] Lustre: 233422:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7171.421979] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7179.461360] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7185.170598] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7193.351055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7193.649287] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7193.688246] Lustre: Permanently deactivating lustre-OST0000 [ 7193.705231] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7201.338584] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7202.392925] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7211.960823] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7221.317156] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7221.371698] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7221.583941] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7221.591274] Lustre: Skipped 11 previous similar messages [ 7228.226934] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7229.116275] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7235.989587] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7237.721623] Lustre: Permanently reactivating lustre-OST0000 [ 7237.733780] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7237.737414] Lustre: Skipped 2 previous similar messages [ 7243.870874] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7243.872724] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 41 seconds. I think it's dead, and I am evicting it. exp 00000000771e5913, cur 1788159076 deadline 1788159065 last 1788159035 [ 7243.880944] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7243.921698] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7243.976266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7252.726368] Lustre: server umount lustre-OST0001 complete [ 7262.987174] Lustre: server umount lustre-OST0000 complete [ 7282.143179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7282.351355] Lustre: server umount lustre-MDT0000 complete [ 7285.747731] LustreError: 234167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159118 with bad export cookie 4218306407972774045 [ 7285.751527] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7285.754391] LustreError: 234167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7286.193644] Lustre: server umount lustre-MDT0001 complete [ 7298.524738] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 7301.512916] Key type lgssc unregistered [ 7301.850979] LNet: 238308:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7301.863082] LNetError: 238308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7301.889077] LNet: Removed LNI 192.168.204.125@tcp [ 7302.953077] Key type .llcrypt unregistered [ 7302.958996] Key type ._llcrypt unregistered [ 7326.409932] Key type ._llcrypt registered [ 7326.417504] Key type .llcrypt registered [ 7326.533659] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 7327.515584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7327.601959] alg: No test for adler32 (adler32-zlib) [ 7328.795468] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7329.091463] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 7330.799253] Key type lgssc registered [ 7332.287090] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7343.339984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7343.364049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7344.942725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7349.403869] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7354.345735] LustreError: 239775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7354.365562] LustreError: 239775:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7358.985451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7359.106889] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7359.149515] Lustre: 239764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7359.161612] Lustre: Permanently deactivating lustre-MDT0001 [ 7359.175349] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7359.448719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7359.905982] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7359.920120] LustreError: 239806:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7359.938206] LustreError: 239806:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7359.955663] LustreError: 239806:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 7364.074281] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7372.745839] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7378.562625] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7385.994231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7386.387522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7389.485589] LustreError: 241617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7389.502051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7393.898722] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7394.814536] LustreError: 241618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7399.923669] LustreError: 241617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7402.549710] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7405.054661] LustreError: 241618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7405.081837] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7405.085721] LustreError: 241618:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7410.470308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7410.680990] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7415.804247] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7415.816047] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7418.627409] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7427.401394] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7431.683563] Lustre: Permanently reactivating lustre-MDT0001 [ 7431.707218] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7431.713262] Lustre: Skipped 2 previous similar messages [ 7437.450635] Lustre: lustre-MDT0001: Client c30bbf7f-3007-438a-9105-dd02edb4de4b (at 192.168.204.25@tcp) reconnecting [ 7445.221615] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7446.755643] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7453.487305] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7453.681501] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7457.640379] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7457.819499] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7460.592589] Lustre: Permanently deactivating lustre-MDT0001 [ 7460.607132] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7460.614208] Lustre: Skipped 2 previous similar messages [ 7469.176273] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7477.222248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7477.241721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7477.248308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7477.250241] Lustre: Skipped 1 previous similar message [ 7477.560408] Lustre: server umount lustre-MDT0000 complete [ 7481.990429] LustreError: 239755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159314 with bad export cookie 12129311486879805150 [ 7481.994256] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7482.007024] LustreError: 239755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7482.339863] Lustre: server umount lustre-MDT0001 complete [ 7488.373166] Lustre: server umount lustre-OST0000 complete [ 7499.474913] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:55:29 (1788159329) [ 7509.506587] Lustre: server umount lustre-OST0001 complete [ 7516.209313] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 7525.910120] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 7565.837995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7566.104851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7566.118994] Lustre: Skipped 2 previous similar messages [ 7566.153105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7566.229991] Lustre: lustre-MDT0000: new disk, initializing [ 7566.331685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7566.361807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7570.635647] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7581.307481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7581.449641] Lustre: 248036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7581.493434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7581.496200] Lustre: Skipped 1 previous similar message [ 7581.558583] Lustre: lustre-MDT0001: new disk, initializing [ 7581.637913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7581.688317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7581.704497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7586.777675] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7597.370923] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7604.255550] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7611.183822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7611.597182] Lustre: lustre-OST0000: new disk, initializing [ 7611.611119] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7611.624560] Lustre: 250005:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7611.731944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7618.695613] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7619.670170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7619.687268] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7619.741201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7629.695361] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7632.123556] LustreError: 248043:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7640.264705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7640.438841] Lustre: lustre-OST0001: new disk, initializing [ 7640.444182] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7640.451289] Lustre: 251118:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7642.159161] LustreError: 248043:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7642.180164] LustreError: 250585:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7646.370729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7646.384210] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7646.599201] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7647.734253] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7652.255836] LustreError: 250585:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7659.916026] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7661.878636] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7664.248327] Lustre: lustre-OST0001: Not available for connect from 192.168.204.25@tcp (stopping) [ 7664.258538] Lustre: Skipped 3 previous similar messages [ 7667.184552] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7667.199535] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7667.200269] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7667.215825] Lustre: Skipped 3 previous similar messages [ 7667.233281] Lustre: Skipped 1 previous similar message [ 7669.327511] Lustre: lustre-OST0001: Not available for connect from 192.168.204.25@tcp (stopping) [ 7670.362397] Lustre: server umount lustre-OST0001 complete [ 7672.298388] LustreError: 249991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7672.332482] LustreError: 249991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7691.746730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7691.747900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7691.765956] LustreError: Skipped 2 previous similar messages [ 7691.790559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7691.800609] Lustre: Skipped 1 previous similar message [ 7692.772169] LustreError: 249991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7692.778382] LustreError: 249991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 7703.008251] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7703.020634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7703.034392] Lustre: Skipped 5 previous similar messages [ 7703.505274] Lustre: server umount lustre-OST0000 complete [ 7709.675273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7709.705153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7709.720371] Lustre: Skipped 1 previous similar message [ 7715.148040] Lustre: server umount lustre-MDT0000 complete [ 7719.081223] LustreError: 251117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159551 with bad export cookie 12129311486879807964 [ 7719.092255] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7719.107088] LustreError: 251117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7725.586174] Lustre: server umount lustre-MDT0001 complete [ 7739.601033] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 7743.473802] Key type lgssc unregistered [ 7743.982438] LNet: 253157:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7744.012607] LNetError: 253157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7745.067893] LNet: Removed LNI 192.168.204.125@tcp [ 7746.189871] Key type .llcrypt unregistered [ 7746.192360] Key type ._llcrypt unregistered [ 7787.568690] Key type ._llcrypt registered [ 7787.570447] Key type .llcrypt registered [ 7787.725479] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 7789.328785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7789.416073] alg: No test for adler32 (adler32-zlib) [ 7790.597220] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7790.890879] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 7792.543264] Key type lgssc registered [ 7793.952668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7802.231698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7802.250588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7803.657593] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7803.699130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7803.922816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7809.036590] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7818.760164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7818.895966] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7818.938295] Lustre: 255365:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7819.220437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7824.382266] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7833.097720] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7839.489717] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7845.964546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7846.148128] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7846.387636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7852.593752] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7854.831923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7860.391471] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7916.513252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7916.528932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7916.536319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7916.548891] Lustre: Skipped 1 previous similar message [ 7917.826115] Lustre: server umount lustre-OST0000 complete [ 7922.178738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7937.001524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7937.462078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7945.208314] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7953.737677] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7956.849894] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7956.866435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7956.869525] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7997.925265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7997.927770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7997.938091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7997.980843] Lustre: Skipped 2 previous similar messages [ 7999.585103] Lustre: server umount lustre-OST0000 complete [ 8008.164671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8008.167739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8008.179949] Lustre: Skipped 1 previous similar message [ 8008.193874] Lustre: Skipped 2 previous similar messages [ 8010.204751] Lustre: server umount lustre-MDT0000 complete [ 8014.134996] LustreError: 255356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159846 with bad export cookie 10219042844067920073 [ 8014.148936] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8014.158707] LustreError: 255356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8014.687645] Lustre: server umount lustre-MDT0001 complete [ 8026.405283] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 8029.303775] Key type lgssc unregistered [ 8029.651663] LNet: 260803:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8029.668696] LNetError: 260803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8029.687896] LNet: Removed LNI 192.168.204.125@tcp [ 8030.543830] Key type .llcrypt unregistered [ 8030.547575] Key type ._llcrypt unregistered [ 8058.990601] Key type ._llcrypt registered [ 8059.004528] Key type .llcrypt registered [ 8059.215708] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 8060.897776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8061.085555] alg: No test for adler32 (adler32-zlib) [ 8062.372854] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8062.713970] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 8064.551244] Key type lgssc registered [ 8065.807897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8074.527208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8074.567550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8076.018651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8081.427787] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8085.472871] LustreError: 262223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8085.482351] LustreError: 262223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8090.600630] LustreError: 262224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8090.909036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8091.334742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8096.789384] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8105.118547] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8112.770585] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8119.359537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8119.836116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8120.898289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8126.920496] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8135.842131] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8166.885461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8166.906207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8166.933120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8167.412533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8167.426400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8167.450926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8167.463094] Lustre: Skipped 1 previous similar message [ 8171.109956] Lustre: server umount lustre-OST0000 complete [ 8177.636283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8177.645192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8177.655918] Lustre: Skipped 1 previous similar message [ 8177.673580] Lustre: Skipped 1 previous similar message [ 8181.627799] Lustre: server umount lustre-MDT0000 complete [ 8182.752307] LustreError: 262218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8182.772921] LustreError: 262218:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8185.449769] LustreError: 262203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160018 with bad export cookie 5003528397021274499 [ 8185.460632] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8185.469583] LustreError: 262203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8185.890508] Lustre: server umount lustre-MDT0001 complete [ 8199.198641] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 8202.090922] Key type lgssc unregistered [ 8202.424659] LNet: 266533:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8202.431487] LNetError: 266533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8202.453450] LNet: Removed LNI 192.168.204.125@tcp [ 8203.333959] Key type .llcrypt unregistered [ 8203.335821] Key type ._llcrypt unregistered [ 8221.433388] Key type ._llcrypt registered [ 8221.436463] Key type .llcrypt registered [ 8221.550119] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 8222.498031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8222.638138] alg: No test for adler32 (adler32-zlib) [ 8223.703401] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8223.960832] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 8225.677101] Key type lgssc registered [ 8227.031502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8235.219974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8235.240264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8236.685951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8241.324826] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8246.243197] LustreError: 267755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8246.256377] LustreError: 267755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8250.071171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8250.441671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8255.996803] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8265.199599] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8271.564395] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8278.649043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8279.040772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8282.105821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8286.199896] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8295.204545] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8307.680523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8307.690829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8307.704408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8310.755160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8310.758645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8310.769733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8310.805529] Lustre: Skipped 1 previous similar message [ 8313.949224] Lustre: server umount lustre-OST0000 complete [ 8320.992588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8321.009968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8324.911666] Lustre: server umount lustre-MDT0000 complete [ 8326.113289] LustreError: 268483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8326.126234] LustreError: 268483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8328.958183] LustreError: 267736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160161 with bad export cookie 3314464778430160544 [ 8328.967940] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8328.983212] LustreError: 267736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8329.356688] Lustre: server umount lustre-MDT0001 complete [ 8342.012944] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 8345.112320] Key type lgssc unregistered [ 8345.428279] LNet: 271433:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8345.450752] LNetError: 271433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8346.475178] LNet: Removed LNI 192.168.204.125@tcp [ 8347.381367] Key type .llcrypt unregistered [ 8347.385859] Key type ._llcrypt unregistered [ 8371.653436] Key type ._llcrypt registered [ 8371.657557] Key type .llcrypt registered [ 8371.762556] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 8372.793374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8372.886644] alg: No test for adler32 (adler32-zlib) [ 8374.057110] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8374.318938] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 8376.031394] Key type lgssc registered [ 8377.128635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8385.556388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8385.581957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8387.163478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8391.667949] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8396.276633] LustreError: 272852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8396.303105] LustreError: 272852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8400.806255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8401.137220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8405.364700] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8412.471542] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8417.587509] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8423.819858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8424.153253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8430.898448] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8431.353688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8438.187662] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8467.429241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8467.432528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8467.442163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8467.445478] Lustre: Skipped 1 previous similar message [ 8470.202769] Lustre: server umount lustre-OST0000 complete [ 8477.664528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8477.669738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8477.672784] Lustre: Skipped 2 previous similar messages [ 8477.684734] Lustre: Skipped 1 previous similar message [ 8481.502822] Lustre: server umount lustre-MDT0000 complete [ 8482.785630] LustreError: 272852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8482.833402] LustreError: 272852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8486.674271] LustreError: 272832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160319 with bad export cookie 10175689999141304800 [ 8486.675683] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8486.691976] LustreError: 272832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8487.188809] Lustre: server umount lustre-MDT0001 complete [ 8499.758830] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 8502.740825] Key type lgssc unregistered [ 8503.185768] LNet: 277160:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8503.193955] LNetError: 277160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8503.221548] LNet: Removed LNI 192.168.204.125@tcp [ 8504.126355] Key type .llcrypt unregistered [ 8504.131823] Key type ._llcrypt unregistered [ 8526.343643] Key type ._llcrypt registered [ 8526.347266] Key type .llcrypt registered [ 8526.495455] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 8528.143714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8528.162070] alg: No test for adler32 (adler32-zlib) [ 8529.307178] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8529.570435] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 8531.288349] Key type lgssc registered [ 8533.132447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8541.715243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8541.754283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8543.493290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8549.548634] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8553.956499] LustreError: 278401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8553.967237] LustreError: 278401:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8559.074914] LustreError: 278403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8559.925310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8560.326553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8565.742152] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8576.017260] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8582.602834] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8589.956242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8590.261638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8597.205912] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8604.996875] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8613.818361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8629.235562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8629.240845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8629.250902] LustreError: 280249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8629.254870] Lustre: Skipped 1 previous similar message [ 8629.385544] Lustre: server umount lustre-OST0000 complete [ 8634.337108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8634.361491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8634.367453] Lustre: Skipped 1 previous similar message [ 8639.457351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8639.470317] Lustre: Skipped 1 previous similar message [ 8644.577080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8644.581304] Lustre: Skipped 1 previous similar message [ 8648.159139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8648.332150] Lustre: server umount lustre-MDT0000 complete [ 8649.699101] LustreError: 278386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8649.707203] LustreError: 278386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8652.534843] LustreError: 278451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160485 with bad export cookie 2955200162544741112 [ 8652.551194] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8652.557064] LustreError: 278451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8652.897886] Lustre: server umount lustre-MDT0001 complete [ 8666.045388] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 8669.678297] Key type lgssc unregistered [ 8670.174517] LNet: 282127:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8670.189164] LNetError: 282127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8670.218376] LNet: Removed LNI 192.168.204.125@tcp [ 8671.634868] Key type .llcrypt unregistered [ 8671.642517] Key type ._llcrypt unregistered [ 8684.696324] Key type ._llcrypt registered [ 8684.699878] Key type .llcrypt registered [ 8684.804835] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8705.106759] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 8722.350292] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 8723.098062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8723.205220] alg: No test for adler32 (adler32-zlib) [ 8724.244630] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8724.455990] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 8726.095321] Key type lgssc registered [ 8727.589153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8769.104332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8769.128946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8770.349413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8770.375910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8770.465435] Lustre: lustre-MDT0000: new disk, initializing [ 8770.553859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8770.579475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8775.707127] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8788.948227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8789.085764] Lustre: 285679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8789.201233] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8789.205251] Lustre: Skipped 1 previous similar message [ 8789.237215] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8789.292250] Lustre: lustre-MDT0001: new disk, initializing [ 8789.387020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8789.430927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8789.447242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8795.043603] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8806.171522] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8812.311921] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8819.396751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8819.840110] Lustre: lustre-OST0000: new disk, initializing [ 8819.845743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8819.853934] Lustre: 287651:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8819.948773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8826.714663] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8829.510819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8829.525517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8829.648989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8836.617510] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8841.414777] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8841.597640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8846.770407] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8846.935627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8850.401645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8850.406209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8850.420844] Lustre: Skipped 1 previous similar message [ 8855.332130] Lustre: server umount lustre-OST0000 complete [ 8860.644144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8860.660871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8860.672517] Lustre: Skipped 2 previous similar messages [ 8865.770512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8865.786159] Lustre: Skipped 1 previous similar message [ 8866.239913] Lustre: server umount lustre-MDT0000 complete [ 8871.082676] LustreError: 286477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160703 with bad export cookie 18002811365635109094 [ 8871.090455] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8871.698613] Lustre: server umount lustre-MDT0001 complete [ 8881.654656] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:18:31 (1788160711) [ 8888.375830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8888.694842] LustreError: 290066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8888.709844] LustreError: 290066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8888.766534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8893.086238] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8893.922457] LustreError: 290067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8899.042802] LustreError: 290066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8902.557068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8902.769659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8908.906216] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8919.510322] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8925.960868] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8932.923212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8933.277641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8941.032429] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8949.815301] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8963.064536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8963.070766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8963.088654] Lustre: Skipped 1 previous similar message [ 8963.090641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8967.668632] Lustre: server umount lustre-OST0000 complete [ 8973.284844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8973.311816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8973.313763] Lustre: Skipped 2 previous similar messages [ 8973.340466] Lustre: Skipped 2 previous similar messages [ 8978.811150] Lustre: server umount lustre-MDT0000 complete [ 8982.508783] LustreError: 290066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8982.536552] LustreError: 290066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8983.218902] LustreError: 290046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160816 with bad export cookie 18002811365635109976 [ 8983.228514] LustreError: 290046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8983.245788] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8983.704052] Lustre: server umount lustre-MDT0001 complete [ 8996.161642] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 8999.570714] Key type lgssc unregistered [ 8999.960458] LNet: 293601:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8999.978320] LNetError: 293601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9000.002435] LNet: Removed LNI 192.168.204.125@tcp [ 9001.013875] Key type .llcrypt unregistered [ 9001.015921] Key type ._llcrypt unregistered [ 9031.341704] Key type ._llcrypt registered [ 9031.344857] Key type .llcrypt registered [ 9031.467822] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 9048.564813] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 9049.594163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9049.784421] alg: No test for adler32 (adler32-zlib) [ 9050.831339] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9051.137613] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 9052.831210] Key type lgssc registered [ 9054.226198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9105.056605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9105.075617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9106.391434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9106.441551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9106.542349] Lustre: lustre-MDT0000: new disk, initializing [ 9106.642541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9106.660585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9111.487326] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9122.176853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9122.339575] Lustre: 297428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9122.426853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9122.438814] Lustre: Skipped 1 previous similar message [ 9122.540738] Lustre: lustre-MDT0001: new disk, initializing [ 9122.690051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9122.776995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9122.785743] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9127.687832] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9138.647187] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9145.616936] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9152.151295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9152.389276] Lustre: lustre-OST2710: new disk, initializing [ 9152.393365] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9152.401212] Lustre: 299397:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9152.472258] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9158.263972] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9160.762840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9160.777220] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9160.845092] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9168.939631] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9176.709148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9176.971718] Lustre: lustre-OST03e8: new disk, initializing [ 9176.978666] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9176.991133] Lustre: 300461:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9177.109900] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9184.030288] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9186.393785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9186.403681] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9186.508307] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9192.714960] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9201.098710] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9201.339979] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9206.246610] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9206.531250] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9212.271659] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9212.604205] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9218.138249] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9218.398813] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9225.200819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9225.210624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9225.486571] Lustre: server umount lustre-MDT0000 complete [ 9227.756679] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9227.776465] Lustre: Skipped 2 previous similar messages [ 9229.563701] LustreError: 297419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161062 with bad export cookie 1408205422360391848 [ 9229.579601] LustreError: 297419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9229.583639] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9229.879229] Lustre: server umount lustre-MDT0001 complete [ 9243.437727] Lustre: server umount lustre-OST2710 complete [ 9257.727540] Lustre: server umount lustre-OST03e8 complete [ 9278.511237] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 9288.018595] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 9326.043676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9326.340164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9326.361771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9326.421402] Lustre: lustre-MDT0000: new disk, initializing [ 9326.468476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9326.481766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9330.673824] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9339.839278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9339.974861] Lustre: 305716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9340.022775] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9340.026632] Lustre: Skipped 1 previous similar message [ 9340.111761] Lustre: lustre-MDT0001: new disk, initializing [ 9340.225133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9340.285112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9340.299972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9344.419158] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9353.734947] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9359.557443] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9366.922557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9367.136971] Lustre: lustre-OST0000: new disk, initializing [ 9367.141107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9367.147286] Lustre: 307683:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9367.230669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9373.899584] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9375.835801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9375.848095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9375.958467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9384.299126] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9390.201435] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9390.535755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9395.569311] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9395.810413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9401.825119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9401.832986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9401.850736] LustreError: Skipped 1 previous similar message [ 9401.856880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9401.865502] Lustre: Skipped 1 previous similar message [ 9401.881270] Lustre: Skipped 1 previous similar message [ 9403.792455] Lustre: server umount lustre-OST0000 complete [ 9412.065196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9412.066630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9412.076077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9412.088377] Lustre: Skipped 1 previous similar message [ 9417.184839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9417.192082] Lustre: Skipped 2 previous similar messages [ 9421.279139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9421.284243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9421.296017] Lustre: Skipped 1 previous similar message [ 9421.490958] Lustre: server umount lustre-MDT0000 complete [ 9425.402406] LustreError: 305707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161258 with bad export cookie 1408205422360393528 [ 9425.407455] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9425.418224] LustreError: 305707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9425.896177] Lustre: server umount lustre-MDT0001 complete [ 9435.264887] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:27:44 (1788161264) [ 9437.258273] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9439.319278] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:27:49 (1788161269) [ 9443.063818] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 9475.190215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9475.473248] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9475.537894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9475.651241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9481.089497] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9491.523578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9491.712818] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9491.758344] Lustre: 311538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9498.683590] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9521.460979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9521.669738] Lustre: Denying initial registration attempt from nid 192.168.204.125@tcp, specified as failover [ 9521.676154] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9521.691311] LustreError: 312994:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -99 [ 9521.698441] LustreError: 312994:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 9521.704682] LustreError: 312994:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9521.758094] Lustre: server umount lustre-OST0000 complete [ 9521.763137] LustreError: 312994:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9531.876442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9531.893722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9531.900821] Lustre: Skipped 1 previous similar message [ 9534.794863] Lustre: server umount lustre-MDT0000 complete [ 9536.997805] LustreError: 312665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9537.018378] LustreError: 312665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9539.006306] LustreError: 313363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161371 with bad export cookie 1408205422360394410 [ 9539.008201] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9539.021135] LustreError: 313363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9539.406834] Lustre: server umount lustre-MDT0001 complete [ 9552.900603] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 9557.815777] Key type lgssc unregistered [ 9558.377406] LNet: 314117:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9558.393720] LNetError: 314117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9558.453587] LNet: Removed LNI 192.168.204.125@tcp [ 9559.748726] Key type .llcrypt unregistered [ 9559.754084] Key type ._llcrypt unregistered [ 9574.017630] Key type ._llcrypt registered [ 9574.022248] Key type .llcrypt registered [ 9574.201955] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 9575.366627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9575.387389] alg: No test for adler32 (adler32-zlib) [ 9576.542629] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9576.889970] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 9578.698210] Key type lgssc registered [ 9579.882477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9629.040564] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 9640.001585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9640.022172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9641.245039] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9641.278655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9641.409179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9647.151798] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9657.515919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9657.635413] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9657.676261] Lustre: 316896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9658.063919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9663.571782] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9672.837784] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9679.145230] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9699.638558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9699.833326] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9700.000487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9707.981364] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9717.176994] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9721.318926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9721.329978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9721.342841] LustreError: Skipped 1 previous similar message [ 9721.366332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9721.372348] Lustre: Skipped 1 previous similar message [ 9721.402103] Lustre: Skipped 1 previous similar message [ 9726.443413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9726.458550] Lustre: Skipped 1 previous similar message [ 9731.558706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9731.578213] Lustre: Skipped 1 previous similar message [ 9734.111252] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9734.372995] Lustre: server umount lustre-OST0000 complete [ 9740.257263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9740.276257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9740.304150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9741.792833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9745.488298] Lustre: server umount lustre-MDT0000 complete [ 9749.976696] LustreError: 317637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161582 with bad export cookie 1983313223340677691 [ 9749.992087] LustreError: 317637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9750.006815] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9750.485133] Lustre: server umount lustre-MDT0001 complete [ 9762.934706] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 9766.008390] Key type lgssc unregistered [ 9766.375864] LNet: 320474:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9766.387229] LNetError: 320474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9766.406371] LNet: Removed LNI 192.168.204.125@tcp [ 9767.291096] Key type .llcrypt unregistered [ 9767.293357] Key type ._llcrypt unregistered [ 9792.379873] Key type ._llcrypt registered [ 9792.382347] Key type .llcrypt registered [ 9792.662277] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 9793.157599] Key type .llcrypt unregistered [ 9793.164860] Key type ._llcrypt unregistered [ 9812.991401] Key type ._llcrypt registered [ 9812.992630] Key type .llcrypt registered [ 9813.108830] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 9814.930201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9815.033781] alg: No test for adler32 (adler32-zlib) [ 9816.302128] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9816.583101] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 9818.287194] Key type lgssc registered [ 9819.661833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9852.886729] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [ 9854.131535] LNet: Added LNI 192.168.204.125@tcp1 [8/256/0/180] [ 9866.584832] loop0: detected capacity change from 0 to 204800000 [ 9866.756975] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9866.772082] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9866.787438] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9884.520567] loop0: detected capacity change from 0 to 204800000 [ 9884.699445] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9884.720214] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9884.742721] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9904.591468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9904.612711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9906.126654] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9906.163726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9906.352852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9911.898274] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9924.767488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9924.848725] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9924.882524] Lustre: 324403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9925.213391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9931.230265] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9942.303617] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9949.767825] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9955.524512] loop0: detected capacity change from 0 to 204800000 [ 9975.277455] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9975.607395] Lustre: sv12302-OST0000: new disk, initializing [ 9975.611843] Lustre: srv-sv12302-OST0000: No data found on store. Initialize space. [ 9975.620108] Lustre: 326484:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9975.675256] Lustre: sv12302-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9983.966787] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9996.562142] loop1: detected capacity change from 0 to 204800000 [10015.862656] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10016.025356] Lustre: sv12302-OST0001: new disk, initializing [10016.029632] Lustre: srv-sv12302-OST0001: No data found on store. Initialize space. [10016.034734] Lustre: 327602:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10016.106449] Lustre: sv12302-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10022.976719] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10036.941146] Lustre: server umount sv12302-OST0001 complete [10047.595855] Lustre: server umount sv12302-OST0000 complete [10064.352815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10064.357378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10064.373910] Lustre: Skipped 1 previous similar message [10069.472852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10069.476815] Lustre: Skipped 2 previous similar messages [10069.658750] Lustre: server umount lustre-MDT0000 complete [10074.286478] LustreError: 325155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161907 with bad export cookie 13813488346650905873 [10074.288549] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10074.297841] LustreError: 325155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10074.957598] Lustre: server umount lustre-MDT0001 complete [10088.378871] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [10092.099699] Key type lgssc unregistered [10092.461578] LNet: 330357:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10092.469454] LNetError: 330357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10092.481653] LNet: Removed LNI 192.168.204.125@tcp [10093.542307] LNet: Removed LNI 192.168.204.125@tcp1 [10094.596315] Key type .llcrypt unregistered [10094.601300] Key type ._llcrypt unregistered [10114.681159] Key type ._llcrypt registered [10114.683021] Key type .llcrypt registered [10114.838403] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [10115.174401] Key type .llcrypt unregistered [10115.180709] Key type ._llcrypt unregistered [10132.781700] Key type ._llcrypt registered [10132.785132] Key type .llcrypt registered [10132.913867] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [10134.270550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10134.308149] alg: No test for adler32 (adler32-zlib) [10135.382798] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10135.617629] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [10137.407186] Key type lgssc registered [10138.680956] Lustre: Echo OBD driver; http://www.lustre.org/ [10163.596525] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [10164.946565] LNet: Added LNI 192.168.204.125@tcp1 [8/256/0/180] [10176.344381] loop0: detected capacity change from 0 to 204800000 [10176.558843] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10176.566188] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10176.574319] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10195.247656] loop0: detected capacity change from 0 to 204800000 [10195.544215] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10195.584983] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10195.596639] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10215.339679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10215.369357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10216.707634] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10216.745420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10216.990599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10221.697800] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10231.485364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10231.612886] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10231.643456] Lustre: 334180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10231.976771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10236.138290] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10243.745373] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10249.419891] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10255.121878] loop0: detected capacity change from 0 to 204800000 [10274.084407] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10274.371287] Lustre: sv8347-OST0000: new disk, initializing [10274.373835] Lustre: srv-sv8347-OST0000: No data found on store. Initialize space. [10274.388896] Lustre: 336260:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10274.500206] Lustre: sv8347-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10280.937894] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10290.514219] loop1: detected capacity change from 0 to 204800000 [10309.372085] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10309.645932] Lustre: sv8347-OST0001: new disk, initializing [10309.652735] Lustre: srv-sv8347-OST0001: No data found on store. Initialize space. [10309.663941] Lustre: 337378:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10309.717819] Lustre: sv8347-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10316.799571] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10329.331583] Lustre: server umount sv8347-OST0001 complete [10337.944416] Lustre: server umount sv8347-OST0000 complete [10353.122562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10353.125760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10353.141026] Lustre: Skipped 1 previous similar message [10353.152090] Lustre: Skipped 1 previous similar message [10356.992766] Lustre: server umount lustre-MDT0000 complete [10361.610205] LustreError: 334175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162194 with bad export cookie 13450323353417985411 [10361.610968] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10361.620480] LustreError: 334175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10362.021672] Lustre: server umount lustre-MDT0001 complete [10374.391601] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [10377.171786] Key type lgssc unregistered [10377.349199] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.25@tcp [10377.376563] LNet: 340133:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10377.384877] LNetError: 340133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10377.394090] LNet: Removed LNI 192.168.204.125@tcp [10378.478596] LNet: Removed LNI 192.168.204.125@tcp1 [10379.297451] Key type .llcrypt unregistered [10379.299167] Key type ._llcrypt unregistered [10403.602258] Key type ._llcrypt registered [10403.607686] Key type .llcrypt registered [10403.853305] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [10404.369520] Key type .llcrypt unregistered [10404.372796] Key type ._llcrypt unregistered [10426.202376] Key type ._llcrypt registered [10426.204033] Key type .llcrypt registered [10426.363952] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [10427.956363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10428.135186] alg: No test for adler32 (adler32-zlib) [10429.209819] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10429.680781] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [10431.487242] Key type lgssc registered [10432.848967] Lustre: Echo OBD driver; http://www.lustre.org/ [10456.872363] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [10458.518472] LNet: Added LNI 192.168.204.125@tcp1 [8/256/0/180] [10468.998301] loop0: detected capacity change from 0 to 204800000 [10469.175343] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10469.182837] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10469.207244] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10486.139649] loop0: detected capacity change from 0 to 204800000 [10486.285773] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10486.298039] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10486.311442] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10503.867677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10503.892369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10505.116036] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10505.127683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10505.232734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10510.442684] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10522.054374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10522.220877] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10522.256199] Lustre: 343959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10522.526494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10527.942419] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10535.150788] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10542.788020] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10550.177325] loop0: detected capacity change from 0 to 204800000 [10570.358457] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10570.871148] Lustre: sv30939-OST0000: new disk, initializing [10570.880723] Lustre: srv-sv30939-OST0000: No data found on store. Initialize space. [10570.891521] Lustre: 346036:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10570.959146] Lustre: sv30939-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10578.313590] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10597.892110] Lustre: server umount sv30939-OST0000 complete [10612.193374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10612.205306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10612.215546] Lustre: Skipped 1 previous similar message [10612.236697] Lustre: Skipped 1 previous similar message [10617.316185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10617.326752] Lustre: Skipped 1 previous similar message [10617.700836] Lustre: server umount lustre-MDT0000 complete [10621.530584] LustreError: 343949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162454 with bad export cookie 11504410114354854256 [10621.545916] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10621.546763] LustreError: 343949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10621.852465] Lustre: server umount lustre-MDT0001 complete [10632.895959] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [10635.930308] Key type lgssc unregistered [10636.256158] LNet: 348493:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10636.263397] LNetError: 348493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10636.272774] LNet: Removed LNI 192.168.204.125@tcp [10637.293354] LNet: Removed LNI 192.168.204.125@tcp1 [10638.370906] Key type .llcrypt unregistered [10638.374207] Key type ._llcrypt unregistered [10658.205556] Key type ._llcrypt registered [10658.208624] Key type .llcrypt registered [10658.318308] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [10658.544493] Key type .llcrypt unregistered [10658.547952] Key type ._llcrypt unregistered [10676.365379] Key type ._llcrypt registered [10676.367513] Key type .llcrypt registered [10676.454593] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [10677.756633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10677.790332] alg: No test for adler32 (adler32-zlib) [10678.931424] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10679.252345] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [10680.927382] Key type lgssc registered [10682.137826] Lustre: Echo OBD driver; http://www.lustre.org/ [10704.617619] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [10706.537799] LNet: Added LNI 192.168.204.125@tcp1 [8/256/0/180] [10718.087516] loop0: detected capacity change from 0 to 204800000 [10718.438063] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10718.468541] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10718.512092] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10736.475093] loop0: detected capacity change from 0 to 204800000 [10736.569180] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10736.585281] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10736.596169] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10756.549187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10756.608395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10758.052464] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10758.084382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10758.286058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10764.100280] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10772.176387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10772.249910] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10772.281263] Lustre: 352366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10772.576240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10775.914521] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10782.530335] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10787.620280] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10792.842566] loop0: detected capacity change from 0 to 204800000 [10810.858678] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10811.085772] Lustre: sv22225-OST0000: new disk, initializing [10811.092253] Lustre: srv-sv22225-OST0000: No data found on store. Initialize space. [10811.099318] Lustre: 354445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10811.157848] Lustre: sv22225-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10817.505103] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10833.849781] Lustre: server umount sv22225-OST0000 complete [10845.155944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10845.189433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10845.201393] Lustre: Skipped 1 previous similar message [10850.274989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10850.284113] Lustre: Skipped 1 previous similar message [10851.546182] Lustre: server umount lustre-MDT0000 complete [10855.956836] LustreError: 352361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162688 with bad export cookie 16074620224635095483 [10855.975855] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10855.980185] LustreError: 352361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10856.326459] Lustre: server umount lustre-MDT0001 complete [10869.020559] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [10872.252529] Key type lgssc unregistered [10872.594229] LNet: 356898:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10872.603605] LNetError: 356898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10872.609285] LNet: Removed LNI 192.168.204.125@tcp [10873.657786] LNet: Removed LNI 192.168.204.125@tcp1 [10874.430990] Key type .llcrypt unregistered [10874.438500] Key type ._llcrypt unregistered [10896.226685] Key type ._llcrypt registered [10896.237305] Key type .llcrypt registered [10896.393775] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [10896.793513] Key type .llcrypt unregistered [10896.797538] Key type ._llcrypt unregistered [10913.311850] Key type ._llcrypt registered [10913.313813] Key type .llcrypt registered [10913.389932] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [10914.712533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10914.743049] alg: No test for adler32 (adler32-zlib) [10915.811670] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10916.053824] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [10917.743111] Key type lgssc registered [10918.849371] Lustre: Echo OBD driver; http://www.lustre.org/ [10939.872255] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [10941.308436] LNet: Added LNI 192.168.204.125@tcp1 [8/256/0/180] [10952.724276] loop0: detected capacity change from 0 to 204800000 [10953.010193] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10953.029105] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10953.049204] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10973.142914] loop0: detected capacity change from 0 to 204800000 [10973.434632] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10973.510482] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10973.544968] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10991.069816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10991.096337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10992.469482] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10992.501516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10992.664179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10996.661328] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11004.933459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11005.021421] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11005.053941] Lustre: 360722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11005.310732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11009.259565] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11015.415700] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11020.752524] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11025.209558] loop0: detected capacity change from 0 to 204800000 [11041.702087] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11041.986391] Lustre: sv5440-OST0000: new disk, initializing [11041.992178] Lustre: srv-sv5440-OST0000: No data found on store. Initialize space. [11041.997993] Lustre: 362798:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11042.097642] Lustre: sv5440-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11048.459547] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11059.553305] loop1: detected capacity change from 0 to 204800000 [11079.436768] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11079.797116] Lustre: sv5440-OST0001: new disk, initializing [11079.804669] Lustre: srv-sv5440-OST0001: No data found on store. Initialize space. [11079.820263] Lustre: 363968:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11079.948279] Lustre: sv5440-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11090.136821] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11105.603670] Lustre: server umount sv5440-OST0001 complete [11114.864464] Lustre: server umount sv5440-OST0000 complete [11129.826252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11129.827734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11129.847405] Lustre: Skipped 1 previous similar message [11129.865620] Lustre: Skipped 1 previous similar message [11134.866747] Lustre: server umount lustre-MDT0000 complete [11138.386117] LustreError: 363967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162971 with bad export cookie 16745209205195177600 [11138.388237] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11138.398101] LustreError: 363967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11138.773745] Lustre: server umount lustre-MDT0001 complete [11150.214336] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [11152.941837] Key type lgssc unregistered [11153.255212] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.25@tcp [11153.297682] LNet: 366723:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11153.309874] LNetError: 366723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11153.319952] LNet: Removed LNI 192.168.204.125@tcp [11154.344282] LNet: Removed LNI 192.168.204.125@tcp1 [11155.619809] Key type .llcrypt unregistered [11155.623930] Key type ._llcrypt unregistered [11172.924585] Key type ._llcrypt registered [11172.927918] Key type .llcrypt registered [11173.016938] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [11173.215176] Key type .llcrypt unregistered [11173.217353] Key type ._llcrypt unregistered [11189.732153] Key type ._llcrypt registered [11189.733787] Key type .llcrypt registered [11189.835936] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [11191.054975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11191.138271] alg: No test for adler32 (adler32-zlib) [11192.348585] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11192.639881] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [11194.408149] Key type lgssc registered [11195.676940] Lustre: Echo OBD driver; http://www.lustre.org/ [11217.104938] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [11218.311814] LNet: Added LNI 192.168.204.125@tcp1 [8/256/0/180] [11251.085694] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [11253.836446] Key type lgssc unregistered [11254.061931] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.25@tcp [11254.118174] LNet: 370687:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11254.133867] LNetError: 370687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11254.144556] LNet: Removed LNI 192.168.204.125@tcp [11254.170185] LNet: Removed LNI 192.168.204.125@tcp1 [11255.275101] Key type .llcrypt unregistered [11255.284532] Key type ._llcrypt unregistered [11274.688033] Key type ._llcrypt registered [11274.690049] Key type .llcrypt registered [11274.789095] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [11275.062896] Key type .llcrypt unregistered [11275.066275] Key type ._llcrypt unregistered [11289.646971] Key type ._llcrypt registered [11289.651973] Key type .llcrypt registered [11289.815843] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [11290.887369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11290.984742] alg: No test for adler32 (adler32-zlib) [11292.013485] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11292.241685] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [11293.935369] Key type lgssc registered [11294.840255] Lustre: Echo OBD driver; http://www.lustre.org/ [11318.140025] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [11320.022424] LNet: Added LNI 192.168.204.125@tcp1 [8/256/0/180] [11322.791316] LNet: Added LNI 192.168.204.125@tcp2 [8/256/0/180] [11337.568267] loop0: detected capacity change from 0 to 204800000 [11337.754696] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11337.765451] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11337.777290] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11355.641409] loop0: detected capacity change from 0 to 204800000 [11355.821753] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11355.833452] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11355.847795] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11375.839814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11375.882443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11377.118428] LNetError: 372009:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.204.125@tcp NID 192.168.204.125@tcp is already locked with peer 192.168.204.125@tcp [11377.250825] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.125@tcp [11377.397870] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11377.472975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11377.758864] LNetError: 372009:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.204.125@tcp NID 192.168.204.125@tcp is already locked with peer 192.168.204.125@tcp [11377.773090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11382.483831] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11393.610894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11393.776230] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11393.817721] Lustre: 374596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11394.080289] LNetError: 372009:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.204.125@tcp NID 192.168.204.125@tcp is already locked with peer 192.168.204.125@tcp [11394.253584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11399.171308] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11399.736628] LNetError: 372009:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.204.125@tcp NID 192.168.204.125@tcp is already locked with peer 192.168.204.125@tcp [11399.749782] LNetError: 372009:0:(peer.c:2151:lnet_peer_add_nid()) Skipped 1 previous similar message [11408.680382] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11414.835430] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11420.660589] loop0: detected capacity change from 0 to 204800000 [11439.402926] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11439.675857] Lustre: sv14512-OST0000: new disk, initializing [11439.679491] Lustre: srv-sv14512-OST0000: No data found on store. Initialize space. [11439.687745] Lustre: 376689:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11439.751901] Lustre: sv14512-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11445.827897] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11457.528827] loop1: detected capacity change from 0 to 204800000 [11475.814365] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11476.101082] Lustre: sv14512-OST0001: new disk, initializing [11476.105329] Lustre: srv-sv14512-OST0001: No data found on store. Initialize space. [11476.112864] Lustre: 377855:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11476.191310] Lustre: sv14512-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11482.651162] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11496.401412] Lustre: server umount sv14512-OST0001 complete [11506.518744] Lustre: server umount sv14512-OST0000 complete [11522.530335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11522.530973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11522.538524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11522.552674] Lustre: Skipped 1 previous similar message [11527.649502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11527.653421] Lustre: Skipped 2 previous similar messages [11532.768628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11532.776267] Lustre: Skipped 1 previous similar message [11534.303932] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11534.651424] Lustre: server umount lustre-MDT0000 complete [11538.506495] LustreError: 374591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163371 with bad export cookie 7658773630005322437 [11538.512670] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11538.522357] LustreError: 374591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11538.946456] Lustre: server umount lustre-MDT0001 complete [11553.179229] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [11556.147432] Key type lgssc unregistered [11556.499199] LNet: 380611:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11556.504644] LNetError: 380611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11556.508727] LNet: Removed LNI 192.168.204.125@tcp [11557.535249] LNet: Removed LNI 192.168.204.125@tcp1 [11557.548244] LNetError: 380611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11557.561987] LNetError: 380611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11557.592054] LNet: Removed LNI 192.168.204.125@tcp2 [11558.781515] Key type .llcrypt unregistered [11558.783193] Key type ._llcrypt unregistered [11578.325746] Key type ._llcrypt registered [11578.328118] Key type .llcrypt registered [11578.425872] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [11578.668311] Key type .llcrypt unregistered [11578.672104] Key type ._llcrypt unregistered [11593.705811] Key type ._llcrypt registered [11593.708474] Key type .llcrypt registered [11593.817204] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [11594.434286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11594.546537] alg: No test for adler32 (adler32-zlib) [11595.677602] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11595.919995] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [11597.567481] Key type lgssc registered [11599.347709] Lustre: Echo OBD driver; http://www.lustre.org/ [11622.053576] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [11623.410307] LNet: Added LNI 192.168.204.125@tcp1 [8/256/0/180] [11634.855566] loop0: detected capacity change from 0 to 204800000 [11635.108557] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11635.136470] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11635.152040] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11652.554717] loop0: detected capacity change from 0 to 204800000 [11652.774085] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11652.793623] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11652.811936] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11671.523689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11671.547563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11672.880098] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11672.908872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11673.030744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11677.679514] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11687.121865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11687.201202] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11687.219710] Lustre: 384483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11687.420251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11693.360239] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11703.311602] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11709.501931] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11715.509737] loop0: detected capacity change from 0 to 204800000 [11735.153966] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11735.401333] Lustre: sv2670-OST0000: new disk, initializing [11735.407377] Lustre: srv-sv2670-OST0000: No data found on store. Initialize space. [11735.418670] Lustre: 386562:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11735.509960] Lustre: sv2670-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11742.646906] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11753.695981] loop1: detected capacity change from 0 to 204800000 [11773.162129] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11773.391809] Lustre: sv2670-OST0001: new disk, initializing [11773.400626] Lustre: srv-sv2670-OST0001: No data found on store. Initialize space. [11773.409517] Lustre: 387679:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11773.497247] Lustre: sv2670-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11781.076826] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11795.887451] Lustre: server umount sv2670-OST0001 complete [11805.141909] Lustre: server umount sv2670-OST0000 complete [11820.517953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11820.536781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11820.573248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11821.543426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11821.566813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11825.116412] Lustre: server umount lustre-MDT0000 complete [11829.456423] LustreError: 384475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163662 with bad export cookie 2053684983152751457 [11829.457510] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11829.477080] LustreError: 384475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11829.802660] Lustre: server umount lustre-MDT0001 complete [11842.193862] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [11845.158162] Key type lgssc unregistered [11845.413292] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.25@tcp [11845.449689] LNet: 390434:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11845.455764] LNetError: 390434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11845.464281] LNet: Removed LNI 192.168.204.125@tcp [11846.501409] LNet: Removed LNI 192.168.204.125@tcp1 [11847.256818] Key type .llcrypt unregistered [11847.259453] Key type ._llcrypt unregistered [11866.670564] Key type ._llcrypt registered [11866.672711] Key type .llcrypt registered [11866.789833] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [11867.142444] Key type .llcrypt unregistered [11867.150337] Key type ._llcrypt unregistered [11873.046723] Key type ._llcrypt registered [11873.050951] Key type .llcrypt registered [11873.159347] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [11873.409913] Key type .llcrypt unregistered [11873.415712] Key type ._llcrypt unregistered [11886.871714] Key type ._llcrypt registered [11886.875516] Key type .llcrypt registered [11886.992810] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [11887.278630] Key type .llcrypt unregistered [11887.285280] Key type ._llcrypt unregistered [11907.511954] Key type ._llcrypt registered [11907.515437] Key type .llcrypt registered [11907.654452] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [11907.920544] Key type .llcrypt unregistered [11907.925743] Key type ._llcrypt unregistered [11924.797664] Key type ._llcrypt registered [11924.800825] Key type .llcrypt registered [11924.970922] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [11926.116357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11926.283702] alg: No test for adler32 (adler32-zlib) [11927.340512] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11927.580655] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [11929.343201] Key type lgssc registered [11930.955196] Lustre: Echo OBD driver; http://www.lustre.org/ [11954.797538] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [11956.274213] LNet: Added LNI 192.168.204.125@tcp1 [8/256/0/180] [11969.408901] loop0: detected capacity change from 0 to 204800000 [11969.696885] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11969.726375] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11969.756536] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11989.015453] loop0: detected capacity change from 0 to 204800000 [11989.352572] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11989.401889] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11989.433171] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12008.900762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12008.927889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12010.317656] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12010.333114] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12010.629750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12016.005210] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12025.890964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12025.983632] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12026.012925] Lustre: 395892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12026.232855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12031.168048] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12039.270547] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12045.066625] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12050.378166] loop0: detected capacity change from 0 to 204800000 [12068.788225] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12069.260957] Lustre: sv24433-OST0000: new disk, initializing [12069.265567] Lustre: srv-sv24433-OST0000: No data found on store. Initialize space. [12069.272886] Lustre: 397970:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12069.321680] Lustre: sv24433-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12075.860704] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12086.485815] loop1: detected capacity change from 0 to 204800000 [12106.412019] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12106.519216] Lustre: sv24433-OST0001: new disk, initializing [12106.530404] Lustre: srv-sv24433-OST0001: No data found on store. Initialize space. [12106.541813] Lustre: 399088:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12106.587783] Lustre: sv24433-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12113.294937] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12129.772437] Lustre: 400078:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12129.784427] Lustre: 400078:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12129.794430] Lustre: 400078:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12129.804234] Lustre: 400078:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12129.809752] Lustre: 400078:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12129.817658] Lustre: 400078:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12131.963966] Lustre: server umount sv24433-OST0001 complete [12141.401461] Lustre: server umount sv24433-OST0000 complete [12154.337486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12154.361087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12154.385918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12155.363756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12155.367461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12160.443808] Lustre: server umount lustre-MDT0000 complete [12164.529880] LustreError: 400176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163997 with bad export cookie 825458653443493278 [12164.546096] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12164.548297] LustreError: 400176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12164.756335] Lustre: server umount lustre-MDT0001 complete [12179.682193] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [12183.135260] Key type lgssc unregistered [12183.514187] LNet: 402086:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12183.521318] LNetError: 402086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12184.545301] LNet: Removed LNI 192.168.204.125@tcp [12184.547310] LNetError: 402086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12184.582061] LNet: Removed LNI 192.168.204.125@tcp1 [12185.568738] Key type .llcrypt unregistered [12185.572303] Key type ._llcrypt unregistered [12206.436805] Key type ._llcrypt registered [12206.439805] Key type .llcrypt registered [12206.547838] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [12206.806432] Key type .llcrypt unregistered [12206.809758] Key type ._llcrypt unregistered [12222.993093] Key type ._llcrypt registered [12222.995246] Key type .llcrypt registered [12223.134806] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [12224.729081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12224.838074] alg: No test for adler32 (adler32-zlib) [12226.060211] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12226.401739] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [12228.145806] Key type lgssc registered [12229.393521] Lustre: Echo OBD driver; http://www.lustre.org/ [12253.155858] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [12254.446839] LNet: Added LNI 192.168.204.125@tcp1 [8/256/0/180] [12265.792386] loop0: detected capacity change from 0 to 204800000 [12266.048605] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12266.061656] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12266.090832] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12283.927616] loop0: detected capacity change from 0 to 204800000 [12284.118146] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12284.136349] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12284.149507] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12301.985986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12302.030222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12303.408165] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12303.451526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12303.643551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12308.169673] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12317.540515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12317.659718] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12317.743205] Lustre: 405909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12318.061730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12323.393895] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12331.799613] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12338.256754] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12343.510690] loop0: detected capacity change from 0 to 204800000 [12362.159425] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12362.600202] Lustre: sv28109-OST0000: new disk, initializing [12362.605433] Lustre: srv-sv28109-OST0000: No data found on store. Initialize space. [12362.614523] Lustre: 407987:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12362.672642] Lustre: sv28109-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12368.941444] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12378.426414] loop1: detected capacity change from 0 to 204800000 [12396.713442] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12396.877606] Lustre: sv28109-OST0001: new disk, initializing [12396.885154] Lustre: srv-sv28109-OST0001: No data found on store. Initialize space. [12396.905578] Lustre: 409104:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12396.981878] Lustre: sv28109-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12403.067296] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12418.095681] Lustre: 410094:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12418.109032] Lustre: 410094:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12418.114920] Lustre: 410094:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12418.120884] Lustre: 410094:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12418.128873] Lustre: 410094:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12418.141437] Lustre: 410094:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12419.632868] Lustre: server umount sv28109-OST0001 complete [12428.782697] Lustre: server umount sv28109-OST0000 complete [12441.057732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12441.068256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12441.077031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12444.137652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12444.143351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12444.171262] Lustre: Skipped 1 previous similar message [12449.249251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12449.260420] Lustre: Skipped 1 previous similar message [12454.370831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12454.379555] Lustre: Skipped 1 previous similar message [12455.391123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12455.548272] Lustre: server umount lustre-MDT0000 complete [12458.874498] LustreError: 405900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164291 with bad export cookie 14613086289169845639 [12458.879255] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12458.884712] LustreError: 405900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12459.201109] Lustre: server umount lustre-MDT0001 complete [12470.872514] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [12473.364718] Key type lgssc unregistered [12473.619564] LNet: 412101:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12473.629108] LNetError: 412101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12473.634553] LNet: Removed LNI 192.168.204.125@tcp [12474.661895] LNet: Removed LNI 192.168.204.125@tcp1 [12475.569057] Key type .llcrypt unregistered [12475.571405] Key type ._llcrypt unregistered [12491.823951] Key type ._llcrypt registered [12491.827273] Key type .llcrypt registered [12491.922307] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [12492.184669] Key type .llcrypt unregistered [12492.187787] Key type ._llcrypt unregistered [12504.075773] Key type ._llcrypt registered [12504.077637] Key type .llcrypt registered [12504.166987] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [12505.018139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12505.032961] alg: No test for adler32 (adler32-zlib) [12506.007705] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12506.166938] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [12507.791173] Key type lgssc registered [12508.538880] Lustre: Echo OBD driver; http://www.lustre.org/ [12531.154588] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [12532.651100] LNet: Added LNI 192.168.204.125@tcp1 [8/256/0/180] [12541.867985] loop0: detected capacity change from 0 to 204800000 [12542.103346] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12542.132416] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12542.155721] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12559.207516] loop0: detected capacity change from 0 to 204800000 [12559.331778] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12559.342083] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12559.350784] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12579.834454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12579.856636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12581.279453] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12581.306959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12581.588511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12585.750475] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12593.980701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12594.178231] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12594.225716] Lustre: 415903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12594.502607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12597.857021] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12603.814672] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12610.179859] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12615.149846] loop0: detected capacity change from 0 to 204800000 [12633.197505] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12633.743080] Lustre: sv5360-OST0000: new disk, initializing [12633.751316] Lustre: srv-sv5360-OST0000: No data found on store. Initialize space. [12633.767822] Lustre: 417981:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12633.908104] Lustre: sv5360-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12643.094409] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12654.242439] loop1: detected capacity change from 0 to 204800000 [12673.834172] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12674.068382] Lustre: sv5360-OST0001: new disk, initializing [12674.074824] Lustre: srv-sv5360-OST0001: No data found on store. Initialize space. [12674.085196] Lustre: 419100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12674.183303] Lustre: sv5360-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12680.655759] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12699.639524] Lustre: 420187:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12699.652459] Lustre: 420187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12699.659423] Lustre: 420187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12699.667254] Lustre: 420187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12699.681273] Lustre: 420187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12699.690708] Lustre: 420187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12700.738367] Lustre: 420235:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12700.747444] Lustre: 420235:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 255 previous similar messages [12700.756552] Lustre: 420235:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12700.764699] Lustre: 420235:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 255 previous similar messages [12700.774578] Lustre: 420235:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12700.780153] Lustre: 420235:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 255 previous similar messages [12700.786838] Lustre: 420235:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12700.792220] Lustre: 420235:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 255 previous similar messages [12700.799865] Lustre: 420235:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12700.809109] Lustre: 420235:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 255 previous similar messages [12700.820069] Lustre: 420235:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12700.826539] Lustre: 420235:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 255 previous similar messages [12703.276558] Lustre: server umount sv5360-OST0001 complete [12712.394270] Lustre: server umount sv5360-OST0000 complete [12727.776193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12727.795342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12727.816097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12729.320109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12729.326196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12731.842637] Lustre: server umount lustre-MDT0000 complete [12736.023367] LustreError: 415895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164568 with bad export cookie 9894705442634712263 [12736.031070] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12736.043387] LustreError: 415895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12736.493232] Lustre: server umount lustre-MDT0001 complete [12750.262585] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [12752.861447] Key type lgssc unregistered [12753.111019] LNet: 422241:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12753.118127] LNetError: 422241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12753.133373] LNet: Removed LNI 192.168.204.125@tcp [12754.216162] LNet: Removed LNI 192.168.204.125@tcp1 [12754.867808] Key type .llcrypt unregistered [12754.870334] Key type ._llcrypt unregistered [12775.519524] Key type ._llcrypt registered [12775.523339] Key type .llcrypt registered [12775.677785] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [12776.049279] Key type .llcrypt unregistered [12776.054667] Key type ._llcrypt unregistered [12791.065548] Key type ._llcrypt registered [12791.067625] Key type .llcrypt registered [12791.163690] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [12791.916792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12791.936771] alg: No test for adler32 (adler32-zlib) [12792.914904] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12793.102469] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [12794.743170] Key type lgssc registered [12795.809671] Lustre: Echo OBD driver; http://www.lustre.org/ [12820.451377] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [12822.164437] LNet: Added LNI 192.168.204.125@tcp1 [8/256/0/180] [12833.457515] loop0: detected capacity change from 0 to 204800000 [12833.695841] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12833.719977] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12833.759184] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12852.124603] loop0: detected capacity change from 0 to 204800000 [12852.261244] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12852.282816] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12852.297105] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12870.399533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12870.416517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12871.717552] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12871.751210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12871.992217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12876.659624] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12884.897235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12885.065513] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12885.138586] Lustre: 426062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12885.612615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12890.481874] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12897.967530] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12904.376474] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12909.883330] loop0: detected capacity change from 0 to 204800000 [12928.943693] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12929.211314] Lustre: sv10339-OST0000: new disk, initializing [12929.220940] Lustre: srv-sv10339-OST0000: No data found on store. Initialize space. [12929.250649] Lustre: 428143:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12929.347891] Lustre: sv10339-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12935.346139] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12950.688248] Lustre: 429083:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12950.694534] Lustre: 429083:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12950.703216] Lustre: 429083:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12950.717296] Lustre: 429083:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12950.724299] Lustre: 429083:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12950.733267] Lustre: 429083:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12956.328470] Lustre: server umount sv10339-OST0000 complete [12968.928915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12968.932130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12968.944130] Lustre: Skipped 1 previous similar message [12974.052500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12974.063345] Lustre: Skipped 2 previous similar messages [12979.168036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12979.177681] Lustre: Skipped 1 previous similar message [12983.263193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12983.476301] Lustre: server umount lustre-MDT0000 complete [12987.748191] LustreError: 429382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164820 with bad export cookie 6820361791158703868 [12987.761109] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12987.762597] LustreError: 429382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12988.143132] Lustre: server umount lustre-MDT0001 complete [13001.611173] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [13005.063536] Key type lgssc unregistered [13006.461829] LNet: 430792:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13006.480483] LNetError: 430792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13006.495384] LNet: Removed LNI 192.168.204.125@tcp [13007.532590] LNet: Removed LNI 192.168.204.125@tcp1 [13008.560983] Key type .llcrypt unregistered [13008.562804] Key type ._llcrypt unregistered [13028.231725] Key type ._llcrypt registered [13028.235886] Key type .llcrypt registered [13028.415952] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [13028.702472] Key type .llcrypt unregistered [13028.709270] Key type ._llcrypt unregistered [13045.340899] Key type ._llcrypt registered [13045.342913] Key type .llcrypt registered [13045.457948] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [13046.794797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13046.905921] alg: No test for adler32 (adler32-zlib) [13047.945581] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13048.237322] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [13049.959166] Key type lgssc registered [13051.183576] Lustre: Echo OBD driver; http://www.lustre.org/ [13074.891124] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [13076.363960] LNet: Added LNI 192.168.204.125@tcp1 [8/256/0/180] [13078.343459] LNet: Added LNI 192.168.204.125@tcp2 [8/256/0/180] [13089.796507] loop0: detected capacity change from 0 to 204800000 [13090.014212] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13090.020921] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13090.033018] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13107.443509] loop0: detected capacity change from 0 to 204800000 [13107.661528] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13107.673206] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13107.684740] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13126.812553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13126.859412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13128.250549] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13128.284301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13128.483507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13134.603032] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13144.781120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13144.862309] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13144.913898] Lustre: 434717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13145.161200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13150.088866] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13158.906280] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13164.576946] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13170.529961] loop0: detected capacity change from 0 to 204800000 [13189.926164] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13190.180216] Lustre: sv16996-OST0000: new disk, initializing [13190.189455] Lustre: srv-sv16996-OST0000: No data found on store. Initialize space. [13190.193685] Lustre: 436796:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13190.250874] Lustre: sv16996-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13195.234927] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13204.606735] loop1: detected capacity change from 0 to 204800000 [13222.638124] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13222.781507] Lustre: sv16996-OST0001: new disk, initializing [13222.792788] Lustre: srv-sv16996-OST0001: No data found on store. Initialize space. [13222.797677] Lustre: 437915:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13222.844384] Lustre: sv16996-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13227.764759] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13234.949001] Lustre: DEBUG MARKER: SKIP: conf-sanity test_57p need a remote client with no Lustre server role for routing [13235.863452] Lustre: 438662:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13235.874492] Lustre: 438662:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13235.881200] Lustre: 438662:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13235.898978] Lustre: 438662:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13235.907518] Lustre: 438662:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13235.912467] Lustre: 438662:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13237.619491] Lustre: server umount sv16996-OST0001 complete [13246.407464] Lustre: server umount sv16996-OST0000 complete [13261.280805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13261.289042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13261.295789] Lustre: Skipped 1 previous similar message [13261.323859] Lustre: Skipped 1 previous similar message [13264.881878] Lustre: server umount lustre-MDT0000 complete [13268.640943] LustreError: 434710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165101 with bad export cookie 17333100516870637528 [13268.650897] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13268.655468] LustreError: 434710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13269.145381] Lustre: server umount lustre-MDT0001 complete [13279.553618] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [13282.015814] Key type lgssc unregistered [13282.243720] LNet: 440669:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13282.249979] LNetError: 440669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13282.253945] LNet: Removed LNI 192.168.204.125@tcp [13283.295450] LNet: Removed LNI 192.168.204.125@tcp1 [13283.325558] LNetError: 440669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13283.336972] LNetError: 440669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13283.362993] LNet: Removed LNI 192.168.204.125@tcp2 [13284.387983] Key type .llcrypt unregistered [13284.390345] Key type ._llcrypt unregistered [13309.115415] Key type ._llcrypt registered [13309.118081] Key type .llcrypt registered [13309.229462] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [13309.524989] Key type .llcrypt unregistered [13309.527145] Key type ._llcrypt unregistered [13329.067932] Key type ._llcrypt registered [13329.070482] Key type .llcrypt registered [13329.165847] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [13330.783468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13330.846720] alg: No test for adler32 (adler32-zlib) [13332.259231] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13332.715713] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [13334.463214] Key type lgssc registered [13335.904801] Lustre: Echo OBD driver; http://www.lustre.org/ [13359.487568] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [13360.836681] LNet: Added LNI 192.168.204.125@tcp1 [8/256/0/180] [13372.273185] loop0: detected capacity change from 0 to 204800000 [13372.427973] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13372.440343] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13372.452677] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13391.191890] loop0: detected capacity change from 0 to 204800000 [13391.411699] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13391.420407] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13391.431041] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13411.978343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13412.014264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13413.415299] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13413.469310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13413.684496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13419.162707] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13431.669655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13431.813307] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13431.885598] Lustre: 444544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13432.158839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13437.231133] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13447.533397] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13453.979461] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13459.969454] loop0: detected capacity change from 0 to 204800000 [13478.440020] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13478.595574] Lustre: sv24102-OST0000: new disk, initializing [13478.598456] Lustre: srv-sv24102-OST0000: No data found on store. Initialize space. [13478.603385] Lustre: 446619:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13478.681502] Lustre: sv24102-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13486.306305] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13500.017525] loop1: detected capacity change from 0 to 204800000 [13519.715162] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13519.906363] Lustre: sv24102-OST0001: new disk, initializing [13519.921592] Lustre: srv-sv24102-OST0001: No data found on store. Initialize space. [13519.929391] Lustre: 447739:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13520.026760] Lustre: sv24102-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13526.953479] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13538.868422] LustreError: 448631:0:(nodemap_handler.c:262:nodemap_disc_visibility_update_end_nm()) cfs_fail_timeout id 2404 sleeping for 15000ms [13553.967147] LustreError: 448631:0:(nodemap_handler.c:262:nodemap_disc_visibility_update_end_nm()) cfs_fail_timeout id 2404 awake [13562.765801] Lustre: 448925:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13562.772942] Lustre: 448925:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13562.777543] Lustre: 448925:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13562.783610] Lustre: 448925:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13562.792277] Lustre: 448925:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13562.801141] Lustre: 448925:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13564.654345] Lustre: server umount sv24102-OST0001 complete [13573.842983] Lustre: server umount sv24102-OST0000 complete [13584.867430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13584.874761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13584.883927] Lustre: Skipped 1 previous similar message [13590.073526] Lustre: server umount lustre-MDT0000 complete [13593.899998] LustreError: 444535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165426 with bad export cookie 5098236272507224201 [13593.912563] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13593.917426] LustreError: 444535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13594.318731] Lustre: server umount lustre-MDT0001 complete [13603.273936] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [13605.880901] Key type lgssc unregistered [13606.179719] LNet: 450932:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13606.194775] LNetError: 450932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13606.209501] LNet: Removed LNI 192.168.204.125@tcp [13607.293145] LNet: Removed LNI 192.168.204.125@tcp1 [13608.105727] Key type .llcrypt unregistered [13608.109093] Key type ._llcrypt unregistered [13628.642544] Key type ._llcrypt registered [13628.643658] Key type .llcrypt registered [13628.727511] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [13629.026591] Key type .llcrypt unregistered [13629.032509] Key type ._llcrypt unregistered [13645.552363] Key type ._llcrypt registered [13645.561723] Key type .llcrypt registered [13645.744340] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [13647.088738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13647.226401] alg: No test for adler32 (adler32-zlib) [13648.416949] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13648.653473] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [13650.360532] Key type lgssc registered [13651.355779] Lustre: Echo OBD driver; http://www.lustre.org/ [13671.921680] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [13673.220229] LNet: Added LNI 192.168.204.125@tcp1 [8/256/0/180] [13683.300504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13683.316410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13684.547554] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13684.561148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13684.666035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13688.846384] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13697.396762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13697.593655] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13697.633456] Lustre: 454202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13697.899540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13703.625290] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13711.994958] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13717.962583] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13722.606977] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [13722.937859] Lustre: DEBUG MARKER: rpc test_57r: @@@@@@ FAIL: Can't determine interface name for NID 192.168.204.125@tcp1 [13725.069904] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing check_logdir /tmp/test_logs/1788165554 [13733.859774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13733.872512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13733.901220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13736.935371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13736.942504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13739.582378] Lustre: server umount lustre-MDT0000 complete [13743.266279] LustreError: 454196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165576 with bad export cookie 14837758033827879817 [13743.272681] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13743.278439] LustreError: 454196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13743.615946] Lustre: server umount lustre-MDT0001 complete [13760.927029] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [13763.820393] Key type lgssc unregistered [13764.073818] LNet: 457547:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13764.078967] LNetError: 457547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13764.083605] LNet: Removed LNI 192.168.204.125@tcp [13765.098544] LNet: Removed LNI 192.168.204.125@tcp1 [13765.887121] Key type .llcrypt unregistered [13765.890401] Key type ._llcrypt unregistered [13781.863596] Key type ._llcrypt registered [13781.865867] Key type .llcrypt registered [13782.018862] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [13782.928600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13782.961129] alg: No test for adler32 (adler32-zlib) [13784.125919] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13784.419654] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [13786.137416] Key type lgssc registered [13787.373910] Lustre: Echo OBD driver; http://www.lustre.org/ [13793.565054] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [13794.484119] LNet: Added LNI 192.168.204.125@tcp1 [8/256/0/180] [13800.610927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13800.647356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13802.083379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13805.814580] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13812.194124] LustreError: 459030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13812.232447] LustreError: 459030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13813.996343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13814.338785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13818.844363] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13826.398297] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13832.769783] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13840.543641] LNet: Removed LNI 192.168.204.125@tcp1 [13841.781621] Lustre: 460744:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13841.788923] Lustre: 460744:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13841.794368] Lustre: 460744:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13841.802410] Lustre: 460744:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13841.810245] Lustre: 460744:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13841.818120] Lustre: 460744:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13847.419800] Lustre: server umount lustre-MDT0000 complete [13850.081017] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13850.091679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13850.103920] LustreError: 459025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13851.932876] LustreError: 459011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165684 with bad export cookie 13843435772861526909 [13851.935375] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13851.942883] LustreError: 459011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13854.496281] Lustre: server umount lustre-MDT0001 complete [13866.382578] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [13869.142494] Key type lgssc unregistered [13869.571181] LNet: 461748:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13869.607157] LNetError: 461748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13870.631712] LNet: Removed LNI 192.168.204.125@tcp [13871.495495] Key type .llcrypt unregistered [13871.497484] Key type ._llcrypt unregistered [13891.673597] Key type ._llcrypt registered [13891.675255] Key type .llcrypt registered [13891.783277] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [13893.101754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13893.143103] alg: No test for adler32 (adler32-zlib) [13894.207595] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13894.407956] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [13896.095276] Key type lgssc registered [13897.244125] Lustre: Echo OBD driver; http://www.lustre.org/ [13905.488316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13905.542077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13906.993224] LNetError: 462377:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.125@tcp1 has no available nets [13907.059124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13911.314720] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13917.675617] LustreError: 463070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13917.693447] LustreError: 463070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13920.686428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13921.044288] LNetError: 462377:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.125@tcp1 has no available nets [13921.055869] LNetError: 462377:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [13921.208908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13926.262517] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13934.418380] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13940.624345] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13949.998842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13950.198639] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [13950.356184] LNetError: 462377:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.125@tcp1 has no available nets [13950.364987] LNetError: 462377:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [13950.408546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13956.422339] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13960.226332] LNetError: 462377:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.125@tcp1 has no available nets [13960.242532] LNetError: 462377:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [13964.612501] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13976.044476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13976.046473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13976.056329] Lustre: Skipped 1 previous similar message [13977.678449] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.25@tcp (stopping) [13977.685092] Lustre: Skipped 2 previous similar messages [13980.369415] Lustre: server umount lustre-MDT0000 complete [13981.154913] LustreError: 463069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13981.173353] LustreError: 463069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13982.774446] LustreError: 464417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13984.382650] LustreError: 463050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165817 with bad export cookie 16410642292802289057 [13984.396942] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13986.272285] LustreError: 463065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13986.279951] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13986.302280] Lustre: Skipped 1 previous similar message [13986.309299] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13989.286783] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13989.294192] Lustre: Skipped 1 previous similar message [13990.689656] Lustre: server umount lustre-MDT0001 complete [13998.435246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [14007.490899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14009.759158] LustreError: 466658:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [14009.768858] LustreError: 466658:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000f9ace025 x1875027665388288/t0(0) o250->MGC192.168.204.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788165842 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14009.801525] LustreError: 466658:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [14009.849520] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14009.867882] Lustre: MGS: Client ae11b1b5-bc08-4d94-a1fd-56ec11d94c95 (at 0@lo) reconnecting [14009.884088] Lustre: MGC192.168.204.125@tcp: Connection restored to 0@lo (at 0@lo) [14010.307964] LNetError: 462377:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.125@tcp1 has no available nets [14010.318760] LNetError: 462377:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [14010.324809] LustreError: 464900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14010.338568] LustreError: 464900:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14010.447614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14010.567188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [14015.820563] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14019.553695] LustreError: 466672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14019.565768] LustreError: 466672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14020.602329] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14025.661440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14026.052741] LNetError: 462377:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.125@tcp1 has no available nets [14026.062578] LNetError: 462377:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [14026.202276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14031.293268] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14031.339592] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14038.905086] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14040.619921] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14046.577860] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14048.152851] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [14056.417058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14056.422514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14056.454480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14056.934278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14059.875918] Lustre: server umount lustre-OST0000 complete [14067.171955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14067.172934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14067.181557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14067.186571] Lustre: Skipped 2 previous similar messages [14067.189604] Lustre: Skipped 1 previous similar message [14069.491805] Lustre: server umount lustre-MDT0000 complete [14072.300203] LustreError: 466675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14072.308448] LustreError: 466675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [14073.323458] LustreError: 463050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165906 with bad export cookie 16410642292802296295 [14073.325930] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14073.348139] LustreError: 463050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [14073.801422] Lustre: server umount lustre-MDT0001 complete [14088.367255] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [14091.533875] Key type lgssc unregistered [14091.820611] LNet: 469346:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14091.826832] LNetError: 469346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14091.839464] LNet: Removed LNI 192.168.204.125@tcp [14092.698267] Key type .llcrypt unregistered [14092.701391] Key type ._llcrypt unregistered [14120.861403] Key type ._llcrypt registered [14120.865770] Key type .llcrypt registered [14120.980556] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [14121.745187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14121.755355] alg: No test for adler32 (adler32-zlib) [14122.813371] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14123.038523] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [14124.695175] Key type lgssc registered [14125.696282] Lustre: Echo OBD driver; http://www.lustre.org/ [14133.735474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14133.755192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14135.403349] LNetError: 470073:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.125@tcp1 has no available nets [14135.537744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14141.218480] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14146.019563] LustreError: 470765:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14146.042631] LustreError: 470765:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14151.139336] LustreError: 470766:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14151.723429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14152.118132] LNetError: 470073:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.125@tcp1 has no available nets [14152.125588] LNetError: 470073:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [14152.286291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14158.167919] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14166.177418] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14172.696357] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14176.738443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14176.771276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14176.779769] Lustre: Skipped 1 previous similar message [14176.796149] Lustre: Skipped 1 previous similar message [14177.825592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14177.837852] Lustre: Skipped 1 previous similar message [14182.517232] Lustre: server umount lustre-MDT0000 complete [14182.881056] LustreError: 470765:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14182.895629] LustreError: 470765:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14186.764660] LustreError: 470746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166019 with bad export cookie 11916689324419674393 [14186.768791] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14186.776157] LustreError: 470746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14187.184525] Lustre: server umount lustre-MDT0001 complete [14202.587779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14202.799702] LNetError: 470073:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.125@tcp1 has no available nets [14202.807917] LNetError: 470073:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 5 previous similar messages [14203.024816] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [14203.094124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14203.382053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14208.663931] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14218.667563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14218.821204] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [14218.860112] Lustre: 473354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14219.036495] LNetError: 470073:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.125@tcp1 has no available nets [14219.047219] LNetError: 470073:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 1 previous similar message [14219.169823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14224.504450] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14233.629238] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14241.558252] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14252.648578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14252.891389] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [14252.911152] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [14252.930131] LustreError: 475240:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [14252.937331] LustreError: 475240:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [14252.941397] LustreError: 475240:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [14253.023402] Lustre: server umount lustre-OST0000 complete [14253.030893] LustreError: 475240:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [14262.840203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14263.148760] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [14263.393548] LNetError: 470073:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.125@tcp1 has no available nets [14263.405152] LNetError: 470073:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [14263.464518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14270.641754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [14271.672859] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14281.730540] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14290.441090] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14290.570971] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [14290.677246] Lustre: lustre-OST0001: new disk, initializing [14290.683284] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [14290.692940] Lustre: 476782:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14290.773612] LNetError: 470073:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.125@tcp1 has no available nets [14290.784160] LNetError: 470073:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 7 previous similar messages [14290.825107] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14297.298608] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14299.328357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14299.345336] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14299.533481] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14305.689898] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14309.858362] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14309.872903] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [14309.873432] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14309.878920] Lustre: Skipped 1 previous similar message [14314.978325] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [14314.995374] Lustre: Skipped 2 previous similar messages [14315.612937] Lustre: server umount lustre-OST0001 complete [14320.098199] LustreError: 475706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14320.102692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14320.125705] LustreError: 475706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14320.153625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14320.157902] Lustre: Skipped 1 previous similar message [14325.217183] LustreError: 477221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14325.235204] LustreError: 477221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14326.244570] Lustre: server umount lustre-OST0000 complete [14331.872385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14331.877394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14331.884222] Lustre: Skipped 1 previous similar message [14331.888354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14331.890917] Lustre: Skipped 2 previous similar messages [14345.183283] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14345.619579] Lustre: server umount lustre-MDT0000 complete [14345.697931] LustreError: 474101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14345.739447] LustreError: 474101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14349.149842] LustreError: 475722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166181 with bad export cookie 11916689324419674946 [14349.153387] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14349.183568] LustreError: 475722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14349.360294] Lustre: server umount lustre-MDT0001 complete [14361.153887] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [14363.704629] Key type lgssc unregistered [14363.997948] LNet: 478741:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14364.010987] LNetError: 478741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14364.035192] LNet: Removed LNI 192.168.204.125@tcp [14364.985797] Key type .llcrypt unregistered [14364.992386] Key type ._llcrypt unregistered [14443.004633] Key type ._llcrypt registered [14443.007631] Key type .llcrypt registered [14443.185590] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [14460.843508] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [14462.583433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14462.649560] alg: No test for adler32 (adler32-zlib) [14463.663165] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14463.832918] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [14465.455099] Key type lgssc registered [14467.027368] Lustre: Echo OBD driver; http://www.lustre.org/ [14510.972320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14510.992303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14512.271934] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14512.328677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14512.438027] Lustre: lustre-MDT0000: new disk, initializing [14512.544412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14512.571547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14517.227316] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14529.748702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14529.902776] Lustre: 483680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14530.056930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14530.062347] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [14530.070844] Lustre: Skipped 1 previous similar message [14530.137778] Lustre: lustre-MDT0001: new disk, initializing [14530.253735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14530.342045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14530.357412] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14536.252853] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14548.222400] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14554.525442] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14561.849367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14562.139757] Lustre: lustre-OST0000: new disk, initializing [14562.142291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14562.148372] Lustre: 485646:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14562.265661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14568.664945] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14571.596538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14571.608447] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14571.778588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14580.006203] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14585.112354] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14585.439815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14591.108162] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14591.388851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14597.603580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14597.614120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14597.621929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14597.636346] Lustre: Skipped 1 previous similar message [14597.669324] LustreError: Skipped 1 previous similar message [14599.760896] Lustre: server umount lustre-OST0000 complete [14607.329594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14607.341150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14607.355145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14607.360714] Lustre: Skipped 1 previous similar message [14610.275116] Lustre: server umount lustre-MDT0000 complete [14614.401563] LustreError: 483671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166447 with bad export cookie 16328680228664339920 [14614.419177] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14614.889928] Lustre: server umount lustre-MDT0001 complete [14625.209245] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:54:14 (1788166454) [14634.671704] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:54:24 (1788166464) [14644.008148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14644.256311] LustreError: 488404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14644.267669] LustreError: 488404:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14644.335511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14648.483163] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14649.825510] LustreError: 488405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14654.946193] LustreError: 488404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14656.023558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14656.327209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14660.199588] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14668.095219] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14673.535303] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14680.567890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14681.029543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14687.053829] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14695.556717] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14703.827135] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [14705.705408] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [14707.558119] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [14709.393882] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [14711.579295] Lustre: Failing over lustre-MDT0000 [14711.911665] Lustre: server umount lustre-MDT0000 complete [14712.801786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14712.809525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14712.837684] Lustre: Skipped 1 previous similar message [14712.850144] LustreError: 488404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14714.852607] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14718.520872] LustreError: 488400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14718.547191] LustreError: 488400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14728.756846] LustreError: 488399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14728.779098] LustreError: 488399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [14731.168110] Lustre: 481902:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788166547/real 1788166547] req@000000009ec347af x1875028262067456/t0(0) o400->MGC192.168.204.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788166563 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14731.188734] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14732.153875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14742.497791] LustreError: 481898:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000033d78243 x1875028262073472/t0(0) o250->MGC192.168.204.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14742.726874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14742.789936] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [14744.120439] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [14747.439908] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14748.150411] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14748.176168] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [14748.211930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [14757.382558] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14759.327640] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14763.423196] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [14765.359465] Lustre: Failing over lustre-MDT0000 [14765.699398] Lustre: server umount lustre-MDT0000 complete [14768.609290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14768.617133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14768.631326] LustreError: 488399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14768.636722] Lustre: Skipped 2 previous similar messages [14768.679182] LustreError: 488399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [14784.996331] Lustre: 481901:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788166601/real 1788166601] req@000000000b375892 x1875028262089600/t0(0) o400->MGC192.168.204.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788166617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14785.054910] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14785.219409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14794.553571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14794.623192] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [14794.844059] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [14799.614513] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14799.866585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [14799.879469] Lustre: Skipped 2 previous similar messages [14799.932195] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [14799.986506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [14809.899787] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14811.594470] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14820.324285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14820.333110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14820.359350] Lustre: Skipped 3 previous similar messages [14820.376155] Lustre: Skipped 2 previous similar messages [14825.122693] Lustre: server umount lustre-MDT0000 complete [14825.450806] LustreError: 488405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14825.485148] LustreError: 488405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [14831.202641] LustreError: 488385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166664 with bad export cookie 16328680228664342636 [14831.204634] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14831.213583] LustreError: 488385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [14831.669504] Lustre: server umount lustre-MDT0001 complete [14847.317258] Lustre: server umount lustre-OST0000 complete [14864.998203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14865.653527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14870.058986] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14880.096056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14885.810381] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14894.455131] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14901.591469] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14909.657204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14910.959107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [14917.208620] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14925.762250] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14931.424385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14931.440270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14931.451912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14931.461371] Lustre: Skipped 1 previous similar message [14940.652245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14940.674593] Lustre: Skipped 3 previous similar messages [14945.247579] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14945.563712] Lustre: server umount lustre-OST0000 complete [14961.121222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14961.129738] Lustre: Skipped 5 previous similar messages [14964.191252] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14964.362118] Lustre: server umount lustre-MDT0000 complete [14965.221635] LustreError: 494749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14965.243144] LustreError: 494749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [14968.771635] LustreError: 494731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166801 with bad export cookie 16328680228664343378 [14968.801153] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14969.165539] Lustre: server umount lustre-MDT0001 complete [14981.817358] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [14985.110350] Key type lgssc unregistered [14985.372796] LNet: 498284:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14985.384900] LNetError: 498284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14985.413614] LNet: Removed LNI 192.168.204.125@tcp [14986.679686] Key type .llcrypt unregistered [14986.682270] Key type ._llcrypt unregistered [15018.727595] Key type ._llcrypt registered [15018.731436] Key type .llcrypt registered [15018.910149] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [15020.344424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15020.385094] alg: No test for adler32 (adler32-zlib) [15021.475636] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15021.725616] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [15023.391757] Key type lgssc registered [15024.697758] Lustre: Echo OBD driver; http://www.lustre.org/ [15034.197971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15034.228669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15035.666815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15040.792422] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15046.116031] LustreError: 499815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15046.148929] LustreError: 499815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15051.234186] LustreError: 499814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15052.716713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15053.278030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15058.972767] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15067.524298] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15073.611475] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15081.209451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15081.645892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15087.403483] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15094.997867] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15106.226707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [15116.774213] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15116.776133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15116.787417] Lustre: Skipped 2 previous similar messages [15116.795887] Lustre: Skipped 2 previous similar messages [15121.889759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15121.899659] Lustre: Skipped 2 previous similar messages [15122.857254] Lustre: server umount lustre-MDT0000 complete [15127.012269] LustreError: 499815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15127.026300] LustreError: 499815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15127.673991] LustreError: 499793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166960 with bad export cookie 1097863936061613007 [15127.693893] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15128.177791] Lustre: server umount lustre-MDT0001 complete [15143.293465] Lustre: server umount lustre-OST0000 complete [15157.606051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15158.071299] LustreError: 503548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15158.156451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15162.109422] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15163.367296] LustreError: 503549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15171.521627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15171.746727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15177.072575] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15186.675649] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15194.226801] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15204.047629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15204.238877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15211.311172] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15219.620376] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15227.848628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [15243.235930] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15243.237632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15243.243450] Lustre: Skipped 1 previous similar message [15243.259717] Lustre: Skipped 2 previous similar messages [15248.363885] LustreError: 503544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15248.388426] LustreError: 503544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15248.471372] Lustre: server umount lustre-MDT0000 complete [15251.518588] LustreError: 503530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167084 with bad export cookie 1097863936061614120 [15251.521872] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15251.528916] LustreError: 503530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [15251.848914] Lustre: server umount lustre-MDT0001 complete [15266.390478] Lustre: server umount lustre-OST0000 complete [15280.674348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15281.290134] LustreError: 507484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15281.384172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15286.032497] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15294.454120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15298.923833] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15306.233787] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15311.714378] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15318.582085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15318.969993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15318.977979] Lustre: Skipped 1 previous similar message [15325.331327] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15327.164239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [15332.996795] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15339.818319] Lustre: server umount lustre-OST0000 complete [15346.145144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15346.166221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15346.186611] Lustre: Skipped 1 previous similar message [15346.190548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15352.808662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15352.823945] Lustre: Skipped 3 previous similar messages [15352.863210] Lustre: server umount lustre-MDT0000 complete [15356.763374] LustreError: 507467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167189 with bad export cookie 1097863936061615142 [15356.772353] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15356.780610] LustreError: 507467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15367.410481] Lustre: server umount lustre-MDT0001 complete [15380.489179] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [15383.643719] Key type lgssc unregistered [15383.976996] LNet: 511019:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15383.982077] LNetError: 511019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15384.010658] LNet: Removed LNI 192.168.204.125@tcp [15384.824834] Key type .llcrypt unregistered [15384.829953] Key type ._llcrypt unregistered [15413.560145] Key type ._llcrypt registered [15413.561846] Key type .llcrypt registered [15413.663652] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [15414.802692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15414.818161] alg: No test for adler32 (adler32-zlib) [15416.068477] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15416.355554] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [15418.015291] Key type lgssc registered [15419.351491] Lustre: Echo OBD driver; http://www.lustre.org/ [15428.809097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15428.836292] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15428.845093] LustreError: 512461:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [15428.872553] LustreError: 512461:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [15428.883828] LustreError: 512461:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [15428.894571] LustreError: 512461:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [15428.903970] LustreError: 512461:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [15437.087445] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [15437.094465] LustreError: 512859:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [15437.113863] LustreError: 512859:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [15437.122498] LustreError: 512859:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [15437.130163] LustreError: 512859:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [15437.140751] LustreError: 512859:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [15459.749209] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [15462.733107] Key type lgssc unregistered [15463.011521] LNet: 513863:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15463.019816] LNetError: 513863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15463.040989] LNet: Removed LNI 192.168.204.125@tcp [15464.124494] Key type .llcrypt unregistered [15464.127082] Key type ._llcrypt unregistered [15485.490528] Key type ._llcrypt registered [15485.495905] Key type .llcrypt registered [15485.637148] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [15502.181302] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [15503.260249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15503.285417] alg: No test for adler32 (adler32-zlib) [15504.364699] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15504.591649] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [15506.279299] Key type lgssc registered [15507.283823] Lustre: Echo OBD driver; http://www.lustre.org/ [15544.660514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15544.715753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15546.282820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15546.360088] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15546.463073] Lustre: lustre-MDT0000: new disk, initializing [15546.516174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15546.541554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15551.579940] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15562.691588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15562.815312] Lustre: 517023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15562.888465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15562.895173] Lustre: Skipped 1 previous similar message [15563.006923] Lustre: lustre-MDT0001: new disk, initializing [15563.148255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15563.219557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15563.239308] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15567.610424] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15576.115396] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15582.477893] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15589.384733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15589.752446] Lustre: lustre-OST0000: new disk, initializing [15589.756323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15589.766261] Lustre: 518991:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15589.863326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15595.086078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15595.093892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15595.253365] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15596.397420] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15606.125761] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15611.335717] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15611.609101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15616.325238] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15616.508334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15621.088730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15621.094554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15621.101958] LustreError: Skipped 1 previous similar message [15621.104023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15621.124078] Lustre: Skipped 1 previous similar message [15624.670568] Lustre: server umount lustre-OST0000 complete [15629.796125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15629.807071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15629.826085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15629.837156] Lustre: Skipped 1 previous similar message [15631.337338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15631.341672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15635.015748] Lustre: server umount lustre-MDT0000 complete [15638.547323] LustreError: 517017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167471 with bad export cookie 2766713552002352087 [15638.549703] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15638.568026] LustreError: 517017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15638.832547] Lustre: server umount lustre-MDT0001 complete [15648.197746] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [15650.054495] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:11:20 (1788167480) [15658.102475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15658.628395] LustreError: 521450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15658.659832] LustreError: 521450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15658.721163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15663.396662] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15664.097479] LustreError: 521451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15668.194476] LustreError: 521450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15672.348519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15672.828374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15678.185913] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15686.164285] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15691.750806] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15698.701782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15699.110547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15705.258748] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15712.970607] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15720.715488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15720.858164] Lustre: lustre-OST0001: new disk, initializing [15720.862821] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [15720.868419] Lustre: 524292:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15720.951625] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [15727.190147] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15729.362418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [15729.384080] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [15729.515551] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [15737.939612] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15744.992384] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15745.003444] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15745.012178] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [15745.020792] Lustre: Skipped 1 previous similar message [15745.033266] Lustre: Skipped 2 previous similar messages [15748.201224] Lustre: server umount lustre-OST0001 complete [15750.124021] LustreError: 523273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15750.139350] LustreError: 523273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15755.238969] LustreError: 523273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15755.259356] LustreError: 523273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15764.965482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15764.978814] LustreError: Skipped 1 previous similar message [15764.981774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15764.993036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15764.998936] Lustre: Skipped 1 previous similar message [15765.479464] LustreError: 523275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15765.492714] LustreError: 523275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15775.717179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15775.723531] Lustre: Skipped 4 previous similar messages [15779.295845] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15779.489764] Lustre: server umount lustre-OST0000 complete [15785.441710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15785.452207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15785.466560] Lustre: Skipped 1 previous similar message [15789.974345] Lustre: server umount lustre-MDT0000 complete [15791.077230] LustreError: 521445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15791.094734] LustreError: 521445:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [15794.752552] LustreError: 521432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167627 with bad export cookie 2766713552002352962 [15794.771185] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15794.781624] LustreError: 521432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15801.376150] Lustre: server umount lustre-MDT0001 complete [15811.919912] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [15814.554833] Key type lgssc unregistered [15814.887708] LNet: 526279:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15814.894320] LNetError: 526279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15815.921257] LNet: Removed LNI 192.168.204.125@tcp [15817.050260] Key type .llcrypt unregistered [15817.053711] Key type ._llcrypt unregistered [15877.375244] Key type ._llcrypt registered [15877.380254] Key type .llcrypt registered [15877.531837] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [15879.281714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15879.329105] alg: No test for adler32 (adler32-zlib) [15880.453192] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15880.811294] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [15882.543198] Key type lgssc registered [15883.795568] Lustre: Echo OBD driver; http://www.lustre.org/ [15891.843586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15891.857150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15893.124428] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [15893.168111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15893.365126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15897.670231] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15906.321073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15906.408719] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [15906.451320] Lustre: 529084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15906.792747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15911.628179] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15919.131995] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15925.106641] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15927.269960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15927.277117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15927.285966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15929.825603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15929.825958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15929.850277] Lustre: Skipped 1 previous similar message [15933.472194] Lustre: server umount lustre-MDT0000 complete [15937.279088] LustreError: 529076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167770 with bad export cookie 13042786734250336991 [15937.284299] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15937.290352] LustreError: 529076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15937.732117] Lustre: server umount lustre-MDT0001 complete [15947.900385] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:16:17 (1788167777) [15954.922270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15955.434027] LustreError: 531595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15955.452828] LustreError: 531595:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15955.504943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15960.088614] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15960.544229] LustreError: 531596:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15965.674302] LustreError: 531595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15969.064808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15969.560812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15973.978695] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15982.055744] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15988.485955] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15995.718666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15995.948507] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [15996.127430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16002.481870] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16010.961105] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16020.495476] Lustre: Permanently deactivating lustre-OST0000 [16020.505078] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [16028.741799] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16030.645978] LustreError: 534583:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [16030.653470] LustreError: 534583:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [16030.657664] LustreError: 534583:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [16035.349935] LustreError: 534632:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [16035.353835] LustreError: 534632:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [16043.663300] Lustre: server umount lustre-OST0000 complete [16046.432439] LustreError: 534882:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [16046.444952] LustreError: 534882:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [16050.151038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16050.158755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16050.173819] Lustre: Skipped 1 previous similar message [16050.196610] Lustre: Skipped 1 previous similar message [16055.265473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16055.279286] Lustre: Skipped 1 previous similar message [16057.405196] Lustre: server umount lustre-MDT0000 complete [16060.389705] LustreError: 532322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16060.418258] LustreError: 532322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16061.737556] LustreError: 531575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167894 with bad export cookie 13042786734250337572 [16061.745548] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16061.753876] LustreError: 531575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16072.330405] Lustre: server umount lustre-MDT0001 complete [16080.207205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16084.956747] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16087.847713] LustreError: 536040:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [16094.405292] Lustre: 536379:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.125@tcp [16096.595580] Lustre: server umount MGS complete [16108.446851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16108.853860] LustreError: 537102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16108.880225] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16108.888967] Lustre: Skipped 1 previous similar message [16108.937119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16113.021247] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16119.264650] LustreError: 537103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16119.271145] LustreError: 537103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16121.269285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16121.584066] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16125.531760] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16132.131907] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16137.871420] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16139.760351] Lustre: Permanently reactivating lustre-OST0000 [16139.793748] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [16139.801487] Lustre: Skipped 2 previous similar messages [16167.913065] LustreError: 538894:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [16175.140353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16175.478583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16175.485618] Lustre: Skipped 1 previous similar message [16181.847629] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16188.815681] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16196.337391] Lustre: server umount lustre-OST0000 complete [16203.232500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16203.239807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16203.254102] Lustre: Skipped 1 previous similar message [16203.270234] Lustre: Skipped 1 previous similar message [16210.512553] Lustre: server umount lustre-MDT0000 complete [16213.286101] Lustre: 528408:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788168030/real 1788168030] req@00000000a15619af x1875029747924096/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788168046 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16213.323632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16213.339236] LustreError: 537102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16213.353775] LustreError: 537102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16214.568846] LustreError: 537084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168047 with bad export cookie 13042786734250338622 [16214.580177] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16214.584622] LustreError: 537084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16215.004939] Lustre: server umount lustre-MDT0001 complete [16227.048750] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [16229.533809] Key type lgssc unregistered [16229.796097] LNet: 541516:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16229.811250] LNetError: 541516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16229.824693] LNet: Removed LNI 192.168.204.125@tcp [16230.824415] Key type .llcrypt unregistered [16230.826863] Key type ._llcrypt unregistered [16252.337612] Key type ._llcrypt registered [16252.342505] Key type .llcrypt registered [16252.531402] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [16272.198521] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [16273.004834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16273.143146] alg: No test for adler32 (adler32-zlib) [16274.177308] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16274.430864] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [16276.183216] Key type lgssc registered [16277.529206] Lustre: Echo OBD driver; http://www.lustre.org/ [16320.261822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16320.301383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16321.591857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16321.617987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16321.714092] Lustre: lustre-MDT0000: new disk, initializing [16321.825309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16321.843357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16326.711976] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16338.218252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16338.305242] Lustre: 544677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16338.361826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16338.365525] Lustre: Skipped 1 previous similar message [16338.428822] Lustre: lustre-MDT0001: new disk, initializing [16338.525342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16338.571764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16338.591029] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16344.119255] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16353.870549] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16360.133595] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16368.099479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16368.410696] Lustre: lustre-OST0000: new disk, initializing [16368.414696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16368.430512] Lustre: 546647:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16368.526878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16375.474071] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16378.458015] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16378.476726] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16378.644778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16385.797454] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16390.356362] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16390.533889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16395.165602] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16395.326486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16399.339391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16399.341923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16399.344854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16399.353549] Lustre: Skipped 1 previous similar message [16399.386402] LustreError: Skipped 1 previous similar message [16403.713540] Lustre: server umount lustre-OST0000 complete [16409.569352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16409.594823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16409.599111] Lustre: Skipped 1 previous similar message [16414.689697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16414.695482] Lustre: Skipped 2 previous similar messages [16419.809359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16419.816404] Lustre: Skipped 1 previous similar message [16422.367141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16422.659334] Lustre: server umount lustre-MDT0000 complete [16426.526949] LustreError: 544670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168259 with bad export cookie 17477687253496867850 [16426.553252] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16426.893175] Lustre: server umount lustre-MDT0001 complete [16435.895874] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [16437.805778] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:24:27 (1788168267) [16445.057030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16445.474455] LustreError: 549109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16445.494502] LustreError: 549109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16445.596865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16449.902399] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16451.046656] LustreError: 549110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16456.164273] LustreError: 549109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16459.671635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16460.032162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16465.069692] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16474.176836] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16481.164838] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16488.939477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16489.360558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16495.994666] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16504.169497] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16506.414278] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [16509.309563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [16516.065967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16516.072752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16516.087644] Lustre: Skipped 1 previous similar message [16516.095802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16520.160279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16520.172319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16521.563730] Lustre: server umount lustre-OST0000 complete [16526.821201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16526.833509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16526.867497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16526.875256] Lustre: Skipped 2 previous similar messages [16532.529690] Lustre: server umount lustre-MDT0000 complete [16535.524536] LustreError: 549104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16535.563807] LustreError: 549104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16537.362826] LustreError: 549091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168370 with bad export cookie 17477687253496868732 [16537.385804] LustreError: 549091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16537.391530] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16537.962881] Lustre: server umount lustre-MDT0001 complete [16550.162374] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [16553.604797] Key type lgssc unregistered [16554.066459] LNet: 552836:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16554.078989] LNetError: 552836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16554.092826] LNet: Removed LNI 192.168.204.125@tcp [16555.635622] Key type .llcrypt unregistered [16555.644762] Key type ._llcrypt unregistered [16588.813512] Key type ._llcrypt registered [16588.822281] Key type .llcrypt registered [16589.035445] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [16589.401695] Key type .llcrypt unregistered [16589.407789] Key type ._llcrypt unregistered [16611.256503] Key type ._llcrypt registered [16611.261698] Key type .llcrypt registered [16611.472517] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [16612.663512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16612.801044] alg: No test for adler32 (adler32-zlib) [16614.105666] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16614.426140] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [16616.239198] Key type lgssc registered [16617.673674] Lustre: Echo OBD driver; http://www.lustre.org/ [16626.897600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16626.923427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16628.455380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16633.411153] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16638.949758] LustreError: 555053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16638.974455] LustreError: 555053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16643.524973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16643.971637] LustreError: 555054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16644.013287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16649.187970] LustreError: 555053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16649.211170] LustreError: 555053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16650.917858] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16653.280918] LustreError: 555054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16653.299187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [16653.299881] LustreError: 555054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16658.401979] LustreError: 555053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16658.412142] LustreError: 555053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16660.092812] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16668.645670] LustreError: 556046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16668.673244] LustreError: 556046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16669.215476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16669.625665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16674.703216] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16685.025495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16685.043862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16685.059565] Lustre: Skipped 1 previous similar message [16690.153853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16690.161923] Lustre: Skipped 1 previous similar message [16695.277115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16695.295786] Lustre: Skipped 1 previous similar message [16699.359144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16699.616574] Lustre: server umount lustre-OST0000 complete [16705.505781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16705.506923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16705.526234] Lustre: Skipped 1 previous similar message [16705.526542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16705.535869] Lustre: Skipped 2 previous similar messages [16710.631028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16710.650046] Lustre: Skipped 1 previous similar message [16718.303172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16718.619656] Lustre: server umount lustre-MDT0000 complete [16720.869441] LustreError: 555053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16720.893751] LustreError: 555053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16723.079455] LustreError: 555034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168555 with bad export cookie 4918271354452041458 [16723.095715] LustreError: 555034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16723.096984] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16723.756922] Lustre: server umount lustre-MDT0001 complete [16737.627721] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [16741.227649] Key type lgssc unregistered [16741.589475] LNet: 558205:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16741.601756] LNetError: 558205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16741.624611] LNet: Removed LNI 192.168.204.125@tcp [16742.651487] Key type .llcrypt unregistered [16742.653374] Key type ._llcrypt unregistered [16777.897417] Key type ._llcrypt registered [16777.899487] Key type .llcrypt registered [16777.990255] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [16778.702464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16778.721414] alg: No test for adler32 (adler32-zlib) [16779.845423] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16780.029648] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [16781.735421] Key type lgssc registered [16782.947699] Lustre: Echo OBD driver; http://www.lustre.org/ [16792.176195] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16792.216130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16807.904589] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [16813.024469] LustreError: 559598:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.125@tcp: failed processing log, type 4: rc = -110 [16839.650503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16848.473990] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16858.036968] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16866.652377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16867.275338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16867.366761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [16872.422720] LustreError: 559624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16872.464093] LustreError: 559624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16873.064844] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16874.477038] LustreError: 560634:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16874.504059] LustreError: 560634:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16879.596464] LustreError: 560634:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16879.612290] LustreError: 560634:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16883.271992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16883.783350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16889.275568] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16897.282824] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16902.957932] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16913.380084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16913.398749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16913.423524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16914.407409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16914.414336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16914.434053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16917.967166] Lustre: server umount lustre-OST0000 complete [16924.643173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16924.647070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16924.661063] Lustre: Skipped 1 previous similar message [16924.673627] Lustre: Skipped 2 previous similar messages [16928.210919] Lustre: server umount lustre-MDT0000 complete [16929.760199] LustreError: 560639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16929.772595] LustreError: 560639:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16931.786744] LustreError: 559606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168764 with bad export cookie 5280708274323328300 [16931.789138] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16931.793071] LustreError: 559606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16932.178407] Lustre: server umount lustre-MDT0001 complete [16943.513075] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [16946.441183] Key type lgssc unregistered [16946.748070] LNet: 563167:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16946.755423] LNetError: 563167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16946.777927] LNet: Removed LNI 192.168.204.125@tcp [16947.762889] Key type .llcrypt unregistered [16947.765710] Key type ._llcrypt unregistered [16970.462943] Key type ._llcrypt registered [16970.465168] Key type .llcrypt registered [16970.581587] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [16971.322188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16971.345104] alg: No test for adler32 (adler32-zlib) [16972.472328] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16972.766610] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [16974.479181] Key type lgssc registered [16975.590853] Lustre: Echo OBD driver; http://www.lustre.org/ [16984.421655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16984.452294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16986.247038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16991.548386] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16996.837587] LustreError: 564584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16996.867455] LustreError: 564584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17000.274037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17000.588249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17005.478195] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [17013.074577] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17018.615557] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17025.686287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17026.068573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17031.939348] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [17032.236692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [17037.959927] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17042.404755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17042.415465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17042.422834] Lustre: Skipped 1 previous similar message [17042.434090] Lustre: Skipped 2 previous similar messages [17045.537718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17045.544853] Lustre: Skipped 1 previous similar message [17048.581758] Lustre: server umount lustre-MDT0000 complete [17050.548663] LustreError: 566428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.25@tcp arrived at 1788168883 with bad export cookie 8082909900270516467 [17050.560266] LustreError: 566428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17050.593702] LustreError: 565312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17050.615461] LustreError: 565312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17052.418819] LustreError: 564564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168885 with bad export cookie 8082909900270515578 [17052.442284] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17052.446754] LustreError: 564564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17055.713480] LustreError: 564583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17055.716122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17055.738403] LustreError: 564583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17055.771572] Lustre: Skipped 1 previous similar message [17055.775460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17055.781075] Lustre: Skipped 1 previous similar message [17057.760414] LustreError: 565312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17057.797257] LustreError: 565312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17058.742948] Lustre: server umount lustre-OST0000 complete [17062.881200] LustreError: 564583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17062.899433] LustreError: 564583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17068.127353] LustreError: 565314:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002ea3329c x1875030892903936/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [17068.334052] Lustre: server umount lustre-MDT0001 complete [17080.937785] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [17084.661452] Key type lgssc unregistered [17085.028087] LNet: 568269:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17085.033812] LNetError: 568269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17085.055302] LNet: Removed LNI 192.168.204.125@tcp [17085.936742] Key type .llcrypt unregistered [17085.940528] Key type ._llcrypt unregistered [17111.997178] Key type ._llcrypt registered [17112.003164] Key type .llcrypt registered [17112.126087] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [17113.542350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17113.582310] alg: No test for adler32 (adler32-zlib) [17114.619533] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17114.848721] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [17116.591229] Key type lgssc registered [17117.642937] Lustre: Echo OBD driver; http://www.lustre.org/ [17124.867829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17124.904768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17126.307625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17132.084095] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [17136.609057] LustreError: 569689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17136.622741] LustreError: 569689:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17141.731686] LustreError: 569688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17143.728766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17144.404787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17149.725541] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [17159.107360] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17166.306991] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17173.931507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17174.389698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17179.887187] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [17190.117304] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17196.938955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [17200.614180] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [17200.621157] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17200.630375] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [17202.152367] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [17202.152499] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17206.522878] Lustre: server umount lustre-MDT0001 complete [17207.270527] LustreError: 569688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17207.299018] LustreError: 569688:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17212.396947] LustreError: 569689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17212.420819] LustreError: 569689:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17217.504853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17217.509895] LustreError: 569688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17217.522386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17217.524988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17217.524994] Lustre: Skipped 2 previous similar messages [17220.872109] Lustre: server umount lustre-OST0000 complete [17225.384966] Lustre: server umount lustre-MDT0000 complete [17241.356712] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [17245.064681] Key type lgssc unregistered [17245.381730] LNet: 573376:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17245.394822] LNetError: 573376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17245.419328] LNet: Removed LNI 192.168.204.125@tcp [17246.417446] Key type .llcrypt unregistered [17246.420323] Key type ._llcrypt unregistered [17276.512430] Key type ._llcrypt registered [17276.515131] Key type .llcrypt registered [17276.673702] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [17293.683715] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [17295.218080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17295.288430] alg: No test for adler32 (adler32-zlib) [17296.397925] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17296.740084] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [17298.399181] Key type lgssc registered [17299.744823] Lustre: Echo OBD driver; http://www.lustre.org/ [17347.165412] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [17365.630703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17365.714608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17366.951427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17367.003256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17367.130168] Lustre: lustre-MDT0000: new disk, initializing [17367.217666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17367.239867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17373.375780] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [17391.948747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17392.255907] Lustre: lustre-OST0000: new disk, initializing [17392.269290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17392.282430] Lustre: Skipped 1 previous similar message [17392.290462] Lustre: 578416:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17392.431824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17400.299886] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [17400.971458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [17400.984864] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [17401.038612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [17411.181534] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17429.271985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17429.432293] Lustre: 577399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17429.509701] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17429.590095] Lustre: lustre-MDT0001: new disk, initializing [17429.787409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17429.834557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [17429.851775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [17434.113197] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [17443.986863] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [17444.169607] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [17449.434696] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [17449.754564] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [17456.106041] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17456.107504] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [17456.131886] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [17456.147736] Lustre: Skipped 1 previous similar message [17456.726924] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.25@tcp (stopping) [17461.222683] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [17461.236419] Lustre: Skipped 1 previous similar message [17461.361398] Lustre: server umount lustre-MDT0001 complete [17471.468398] LustreError: 577407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17471.480671] LustreError: 577407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [17476.589288] LustreError: 577410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17476.611691] LustreError: 577410:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17481.702479] LustreError: 577411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17481.704816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17481.744276] Lustre: Skipped 1 previous similar message [17481.750527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17486.649266] Lustre: server umount lustre-OST0000 complete [17486.817293] LustreError: 577410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17491.942680] LustreError: 577411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17498.362972] Lustre: server umount lustre-MDT0000 complete [17515.027889] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [17518.469695] Key type lgssc unregistered [17518.786781] LNet: 582164:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17518.796604] LNetError: 582164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17518.819968] LNet: Removed LNI 192.168.204.125@tcp [17520.025177] Key type .llcrypt unregistered [17520.028160] Key type ._llcrypt unregistered [17568.330888] Key type ._llcrypt registered [17568.339963] Key type .llcrypt registered [17568.549773] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [17568.987286] Key type .llcrypt unregistered [17568.991293] Key type ._llcrypt unregistered [17589.164464] Key type ._llcrypt registered [17589.168750] Key type .llcrypt registered [17589.296405] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [17590.791980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17590.858818] alg: No test for adler32 (adler32-zlib) [17591.982454] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17592.321139] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [17594.023721] Key type lgssc registered [17595.496329] Lustre: Echo OBD driver; http://www.lustre.org/ [17627.489566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17627.533392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17628.859371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17628.910930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17629.025587] Lustre: lustre-MDT0000: new disk, initializing [17629.119361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17629.145530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17634.184950] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [17646.966474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17647.146823] Lustre: 585547:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17647.260869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17647.265795] Lustre: Skipped 1 previous similar message [17647.400804] Lustre: lustre-MDT0001: new disk, initializing [17647.549430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17647.616280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17647.633225] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17652.874469] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [17664.233552] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17671.654980] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17679.972789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17680.324726] Lustre: lustre-OST0000: new disk, initializing [17680.328543] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17680.336659] Lustre: 587516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17680.471194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17687.055809] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [17689.147282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17689.160160] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17689.256593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17697.974893] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17709.032361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17709.042824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17709.056867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17710.051037] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17710.055480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17711.952113] Lustre: server umount lustre-MDT0000 complete [17716.285389] LustreError: 586342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169549 with bad export cookie 4209057442131157339 [17716.307902] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17716.322726] LustreError: 586342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [17716.730795] Lustre: server umount lustre-MDT0001 complete [17731.344808] Lustre: server umount lustre-OST0000 complete [17746.399654] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:46:15 (1788169575) [17755.911095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17756.368570] LustreError: 589545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17756.385230] LustreError: 589545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [17756.436572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17761.765664] LustreError: 589546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17761.788557] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [17766.881856] LustreError: 589545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17772.005408] LustreError: 589546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17772.195989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17772.625618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17778.258734] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [17787.993621] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17796.439737] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17816.966221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17817.168198] Lustre: Found index 0 for lustre-OST0000, updating log [17817.338947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17825.093712] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [17826.565472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [17834.989117] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17844.195101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17844.205492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17844.214311] Lustre: Skipped 1 previous similar message [17844.221904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17844.230384] Lustre: Skipped 2 previous similar messages [17850.263416] Lustre: server umount lustre-OST0000 complete [17869.798123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17869.802194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17869.818586] Lustre: Skipped 1 previous similar message [17869.826440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17869.833228] Lustre: Skipped 3 previous similar messages [17873.896807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17873.909057] Lustre: Skipped 4 previous similar messages [17874.136169] Lustre: server umount lustre-MDT0000 complete [17878.169391] LustreError: 589525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169710 with bad export cookie 4209057442131158732 [17878.170930] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17878.177624] LustreError: 589525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [17878.487827] Lustre: server umount lustre-MDT0001 complete [17880.095716] LNet: 1 local NIs in recovery (showing 1): 192.168.204.125@tcp [17889.338437] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:48:38 (1788169718) [18517.458016] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18517.872503] LustreError: 595966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18517.949153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18522.234827] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [18523.106753] LustreError: 595967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18527.204519] LustreError: 595966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18532.803973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18533.038480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18538.601674] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [18548.910253] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18556.235867] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18573.343552] LNet: 1 local NIs in recovery (showing 1): 192.168.204.125@tcp [18578.043519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18578.320953] Lustre: Found index 0 for lustre-OST0000, updating log [18578.621079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18586.627212] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [18595.951556] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18599.536215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18599.553576] Lustre: Skipped 1 previous similar message [18599.561577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18599.571754] Lustre: Skipped 1 previous similar message [18604.768311] Lustre: server umount lustre-OST0000 complete [18610.018834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18610.024615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18610.041963] Lustre: Skipped 1 previous similar message [18610.057121] Lustre: Skipped 1 previous similar message [18612.129353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18612.140834] Lustre: Skipped 3 previous similar messages [18615.271468] Lustre: server umount lustre-MDT0000 complete [18616.241884] LustreError: 595967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18616.262027] LustreError: 595967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [18619.161712] LustreError: 595948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170451 with bad export cookie 4209057442131159817 [18619.167222] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18619.200151] LustreError: 595948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [18619.568400] Lustre: server umount lustre-MDT0001 complete [18633.823454] LNet: 1 local NIs in recovery (showing 1): 192.168.204.125@tcp [18655.909613] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [18667.709525] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [18711.939279] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:02:21 (1788170541) [18732.823248] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [18736.097704] Key type lgssc unregistered [18736.367976] LNet: 602591:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18736.376532] LNetError: 602591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18838.835570] LNet: Removed LNI 192.168.204.125@tcp [18840.000162] Key type .llcrypt unregistered [18840.002269] Key type ._llcrypt unregistered [18857.341897] Key type ._llcrypt registered [18857.344145] Key type .llcrypt registered [18857.435256] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [18859.005817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18859.151107] alg: No test for adler32 (adler32-zlib) [18860.258789] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [18860.558513] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [18862.249844] Key type lgssc registered [18863.739941] Lustre: Echo OBD driver; http://www.lustre.org/ [18869.081873] LNet: Added LNI 192.168.204.125@tcp100 [8/256/0/180] [18870.040679] LNet: Added LNI 192.168.204.125@tcp101 [8/256/0/180] [18871.341451] LNet: Added LNI 192.168.204.125@tcp102 [8/256/0/180] [18872.337359] LNet: Added LNI 192.168.204.125@tcp103 [8/256/0/180] [18873.404358] LNet: Added LNI 192.168.204.125@tcp104 [8/256/0/180] [18874.429971] LNet: Added LNI 192.168.204.125@tcp105 [8/256/0/180] [18875.314527] LNet: Added LNI 192.168.204.125@tcp106 [8/256/0/180] [18876.277825] LNet: Added LNI 192.168.204.125@tcp107 [8/256/0/180] [18877.230164] LNet: Added LNI 192.168.204.125@tcp108 [8/256/0/180] [18878.351590] LNet: Added LNI 192.168.204.125@tcp109 [8/256/0/180] [18879.245713] LNet: Added LNI 192.168.204.125@tcp110 [8/256/0/180] [18880.022723] LNet: Added LNI 192.168.204.125@tcp111 [8/256/0/180] [18880.939651] LNet: Added LNI 192.168.204.125@tcp112 [8/256/0/180] [18881.916064] LNet: Added LNI 192.168.204.125@tcp113 [8/256/0/180] [18882.830920] LNet: Added LNI 192.168.204.125@tcp114 [8/256/0/180] [18883.780787] LNet: Added LNI 192.168.204.125@tcp115 [8/256/0/180] [18884.790305] LNet: Added LNI 192.168.204.125@tcp116 [8/256/0/180] [18885.775089] LNet: Added LNI 192.168.204.125@tcp117 [8/256/0/180] [18886.856270] LNet: Added LNI 192.168.204.125@tcp118 [8/256/0/180] [18887.709895] LNet: Added LNI 192.168.204.125@tcp119 [8/256/0/180] [18888.800883] LNet: Added LNI 192.168.204.125@tcp120 [8/256/0/180] [18889.782306] LNet: Added LNI 192.168.204.125@tcp121 [8/256/0/180] [18890.817692] LNet: Added LNI 192.168.204.125@tcp122 [8/256/0/180] [18891.728607] LNet: Added LNI 192.168.204.125@tcp123 [8/256/0/180] [18892.708910] LNet: Added LNI 192.168.204.125@tcp124 [8/256/0/180] [18893.531710] LNet: Added LNI 192.168.204.125@tcp125 [8/256/0/180] [18894.429222] LNet: Added LNI 192.168.204.125@tcp126 [8/256/0/180] [18895.573191] LNet: Added LNI 192.168.204.125@tcp127 [8/256/0/180] [18896.303914] LNet: Added LNI 192.168.204.125@tcp128 [8/256/0/180] [18897.159384] LNet: Added LNI 192.168.204.125@tcp129 [8/256/0/180] [18898.120929] LNet: Added LNI 192.168.204.125@tcp130 [8/256/0/180] [18899.043050] LNet: Added LNI 192.168.204.125@tcp131 [8/256/0/180] [18899.929957] LNet: Added LNI 192.168.204.125@tcp132 [8/256/0/180] [18900.729478] LNet: Added LNI 192.168.204.125@tcp133 [8/256/0/180] [18901.499834] LNet: Added LNI 192.168.204.125@tcp134 [8/256/0/180] [18902.107648] LNet: Added LNI 192.168.204.125@tcp135 [8/256/0/180] [18909.557867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18909.601366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18910.755123] Lustre: 605623:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.125@tcp: can't add NID 192.168.204.125@tcp130: rc = -75 [18910.883985] LNetError: 605643:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [18910.937487] LNetError: Protocol error connecting to 192.168.204.125@tcp120 at host 192.168.204.125:988: is it running a compatible version of Lustre? [18910.943189] LNetError: Skipped 2 previous similar messages [18911.133848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [18911.192905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [18911.323798] Lustre: lustre-MDT0000: new disk, initializing [18911.434897] Lustre: 605680:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.125@tcp: can't add NID 192.168.204.125@tcp130: rc = -75 [18911.442780] Lustre: 605680:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [18911.468636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18911.494288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [18911.775670] LNet: 5 local NIs in recovery (showing 5): 192.168.204.125@tcp125, 192.168.204.125@tcp126, 192.168.204.125@tcp120, 192.168.204.125@tcp127, 192.168.204.125@tcp128 [18911.785240] LNetError: 605647:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [18911.824219] LNetError: 605647:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [18911.839083] LNetError: Protocol error connecting to 192.168.204.125@tcp at host 192.168.204.125:988: is it running a compatible version of Lustre? [18911.851292] LNetError: Skipped 6 previous similar messages [18916.916839] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [18929.334474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18929.516153] Lustre: 605638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [18929.619566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [18929.624162] Lustre: Skipped 1 previous similar message [18929.712249] Lustre: lustre-MDT0001: new disk, initializing [18929.734741] Lustre: 606444:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.125@tcp: can't add NID 192.168.204.125@tcp130: rc = -75 [18929.739708] Lustre: 606444:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [18929.859621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18929.890148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [18929.912732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [18935.188442] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [18937.943992] Lustre: 606808:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.125@tcp: can't add NID 192.168.204.125@tcp130: rc = -75 [18937.953145] Lustre: 606808:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [18942.497197] LNetError: 605656:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [18942.508372] LNetError: Protocol error connecting to 192.168.204.125@tcp at host 192.168.204.125:988: is it running a compatible version of Lustre? [18945.182839] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18948.640694] LNetError: 605644:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [18948.656677] LNetError: 605644:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [18948.663879] LNetError: Protocol error connecting to 192.168.204.125@tcp124 at host 192.168.204.125:988: is it running a compatible version of Lustre? [18948.676244] LNetError: Skipped 1 previous similar message [18951.524464] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18954.789377] LNetError: 603082:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [18954.824752] LNetError: 603082:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [18954.846235] LNetError: Protocol error connecting to 192.168.204.125@tcp at host 192.168.204.125:988: is it running a compatible version of Lustre? [18954.858478] LNetError: Skipped 1 previous similar message [18958.944846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18959.373328] Lustre: lustre-OST0000: new disk, initializing [18959.377915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [18959.385800] Lustre: 607616:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [18959.540392] Lustre: 607621:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.125@tcp: can't add NID 192.168.204.125@tcp130: rc = -75 [18959.551038] Lustre: 607621:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [18959.600250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18964.014919] LNetError: 605649:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [18964.031568] LNetError: 605649:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [18964.047926] LNetError: Protocol error connecting to 192.168.204.125@tcp122 at host 192.168.204.125:988: is it running a compatible version of Lustre? [18964.070469] LNetError: Skipped 2 previous similar messages [18965.120524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [18965.125737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [18965.273761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [18967.301851] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [18977.593941] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18981.409898] LNetError: 603081:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [18981.416548] LNetError: 603081:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [18981.419902] LNetError: Protocol error connecting to 192.168.204.125@tcp123 at host 192.168.204.125:988: is it running a compatible version of Lustre? [18981.428540] LNetError: Skipped 4 previous similar messages [18985.953303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18985.968745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [18985.971966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18991.097562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18991.104639] Lustre: Skipped 2 previous similar messages [18996.204575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18996.213682] Lustre: Skipped 1 previous similar message [18999.263126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [18999.410242] Lustre: server umount lustre-OST0000 complete [19006.437543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19006.451740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19006.452114] Lustre: Skipped 2 previous similar messages [19006.481676] Lustre: Skipped 1 previous similar message [19008.973823] Lustre: server umount lustre-MDT0000 complete [19013.142505] LustreError: 605631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170845 with bad export cookie 8141649938894168353 [19013.144670] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19013.160148] LustreError: 605631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [19013.555365] Lustre: server umount lustre-MDT0001 complete [19015.200611] LNetError: 603083:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [19015.215630] LNetError: 603083:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [19015.226626] LNetError: Protocol error connecting to 192.168.204.125@tcp122 at host 192.168.204.125:988: is it running a compatible version of Lustre? [19015.240125] LNetError: Skipped 5 previous similar messages [19025.221399] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [19027.826743] Key type lgssc unregistered [19028.222574] LNet: 609530:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19028.230754] LNetError: 609530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19028.239269] LNet: Removed LNI 192.168.204.125@tcp [19029.279444] LNet: Removed LNI 192.168.204.125@tcp100 [19029.288629] LNetError: 609530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19029.302577] LNetError: 609530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [19029.319237] LNet: Removed LNI 192.168.204.125@tcp101 [19030.367352] LNet: Removed LNI 192.168.204.125@tcp102 [19030.372293] LNetError: 609530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19030.382510] LNetError: 609530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [19030.391062] LNet: Removed LNI 192.168.204.125@tcp103 [19031.455179] LNet: Removed LNI 192.168.204.125@tcp104 [19032.479186] LNet: Removed LNI 192.168.204.125@tcp105 [19032.486665] LNetError: 609530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19032.502463] LNetError: 609530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [19032.509183] LNet: Removed LNI 192.168.204.125@tcp106 [19033.567158] LNet: Removed LNI 192.168.204.125@tcp107 [19034.591226] LNet: Removed LNI 192.168.204.125@tcp108 [19035.615185] LNet: Removed LNI 192.168.204.125@tcp109 [19036.639193] LNet: Removed LNI 192.168.204.125@tcp110 [19036.642589] LNetError: 609530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19036.653641] LNetError: 609530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [19036.659911] LNet: Removed LNI 192.168.204.125@tcp111 [19037.727445] LNet: Removed LNI 192.168.204.125@tcp112 [19038.751724] LNet: Removed LNI 192.168.204.125@tcp113 [19039.775164] LNet: Removed LNI 192.168.204.125@tcp114 [19040.800223] LNet: Removed LNI 192.168.204.125@tcp115 [19041.823351] LNet: Removed LNI 192.168.204.125@tcp116 [19042.848723] LNet: Removed LNI 192.168.204.125@tcp117 [19043.871815] LNet: Removed LNI 192.168.204.125@tcp118 [19044.895350] LNet: Removed LNI 192.168.204.125@tcp119 [19044.911399] LNetError: 609530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19044.918072] LNetError: 609530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [19045.985235] LNet: Removed LNI 192.168.204.125@tcp120 [19047.007237] LNet: Removed LNI 192.168.204.125@tcp121 [19048.031399] LNet: Removed LNI 192.168.204.125@tcp122 [19049.058723] LNet: Removed LNI 192.168.204.125@tcp123 [19050.080543] LNet: Removed LNI 192.168.204.125@tcp124 [19051.103693] LNet: Removed LNI 192.168.204.125@tcp125 [19052.127181] LNet: Removed LNI 192.168.204.125@tcp126 [19053.152751] LNet: Removed LNI 192.168.204.125@tcp127 [19054.175163] LNet: Removed LNI 192.168.204.125@tcp128 [19055.199152] LNet: Removed LNI 192.168.204.125@tcp129 [19055.201393] LNet: Removed LNI 192.168.204.125@tcp130 [19055.208187] LNet: Removed LNI 192.168.204.125@tcp131 [19055.212688] LNet: Removed LNI 192.168.204.125@tcp132 [19055.216531] LNet: Removed LNI 192.168.204.125@tcp133 [19055.221620] LNet: Removed LNI 192.168.204.125@tcp134 [19055.245280] LNet: Removed LNI 192.168.204.125@tcp135 [19056.030641] Key type .llcrypt unregistered [19056.035536] Key type ._llcrypt unregistered [19079.167519] Key type ._llcrypt registered [19079.169679] Key type .llcrypt registered [19079.300842] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [19096.139903] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [19097.433337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19097.494837] alg: No test for adler32 (adler32-zlib) [19098.608613] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19098.912866] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [19100.643093] Key type lgssc registered [19101.717515] Lustre: Echo OBD driver; http://www.lustre.org/ [19136.263157] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 06:09:26 (1788170966) [19154.296260] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [19157.429139] Key type lgssc unregistered [19157.779918] LNet: 613619:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19157.792597] LNetError: 613619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19157.807854] LNet: Removed LNI 192.168.204.125@tcp [19158.783731] Key type .llcrypt unregistered [19158.785308] Key type ._llcrypt unregistered [19177.364513] Key type ._llcrypt registered [19177.373755] Key type .llcrypt registered [19177.486438] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [19178.846880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19178.866577] alg: No test for adler32 (adler32-zlib) [19179.926584] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19180.161622] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [19181.863239] Key type lgssc registered [19183.093515] Lustre: Echo OBD driver; http://www.lustre.org/ [19192.313075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19192.398261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19193.672245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [19193.720982] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [19193.791356] Lustre: lustre-MDT0000: new disk, initializing [19193.856412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19193.873476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19199.037730] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [19210.428045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19210.577798] Lustre: 614833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [19210.674287] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [19210.689439] Lustre: Skipped 1 previous similar message [19210.700371] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [19210.768749] Lustre: lustre-MDT0001: new disk, initializing [19210.882817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19210.957501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [19210.972250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [19215.763284] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [19226.093725] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19232.193185] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19238.930487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19239.283733] Lustre: lustre-OST0000: new disk, initializing [19239.287629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [19239.296589] Lustre: 616803:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19239.385277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19245.107960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19245.128576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19245.268811] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19245.339520] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [19253.851949] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19257.817395] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [19264.030749] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [19264.086836] LNet: Added LNI 192.168.204.125@tcp10 [8/256/0/180] [19269.951688] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [19269.991982] LNet: Added LNI 192.168.204.125@tcp11 [8/256/0/180] [19274.116399] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [19274.147546] LNet: Added LNI 192.168.204.125@tcp12 [8/256/0/180] [19278.311085] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [19278.341321] LNet: Added LNI 192.168.204.125@tcp13 [8/256/0/180] [19282.414405] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [19282.450433] LNet: Added LNI 192.168.204.125@tcp14 [8/256/0/180] [19287.513762] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [19287.538638] LNet: Added LNI 192.168.204.125@tcp15 [8/256/0/180] [19292.264443] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [19292.293149] LNet: Added LNI 192.168.204.125@tcp16 [8/256/0/180] [19297.431359] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [19297.469951] LNet: Added LNI 192.168.204.125@tcp17 [8/256/0/180] [19303.009431] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [19303.041472] LNet: Added LNI 192.168.204.125@tcp18 [8/256/0/180] [19308.361542] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [19308.409962] LNet: Added LNI 192.168.204.125@tcp19 [8/256/0/180] [19313.307394] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [19313.345306] LNet: Added LNI 192.168.204.125@tcp20 [8/256/0/180] [19318.738326] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [19318.764413] LNet: Added LNI 192.168.204.125@tcp21 [8/256/0/180] [19325.648270] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [19325.684190] LNet: Added LNI 192.168.204.125@tcp22 [8/256/0/180] [19330.960398] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [19330.994256] LNet: Added LNI 192.168.204.125@tcp23 [8/256/0/180] [19336.235564] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [19336.259194] LNet: Added LNI 192.168.204.125@tcp24 [8/256/0/180] [19342.906354] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [19342.952475] LNet: Added LNI 192.168.204.125@tcp25 [8/256/0/180] [19351.440205] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [19351.474253] LNet: Added LNI 192.168.204.125@tcp26 [8/256/0/180] [19359.536245] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [19359.602839] LNet: Added LNI 192.168.204.125@tcp27 [8/256/0/180] [19366.387638] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [19366.448120] LNet: Added LNI 192.168.204.125@tcp28 [8/256/0/180] [19372.725265] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [19372.762238] LNet: Added LNI 192.168.204.125@tcp29 [8/256/0/180] [19380.282519] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [19380.315300] LNet: Added LNI 192.168.204.125@tcp30 [8/256/0/180] [19387.021287] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [19387.075797] LNet: Added LNI 192.168.204.125@tcp31 [8/256/0/180] [19393.903259] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [19393.949243] LNet: Added LNI 192.168.204.125@tcp32 [8/256/0/180] [19401.086216] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [19401.112651] LNet: Added LNI 192.168.204.125@tcp33 [8/256/0/180] [19408.375869] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [19408.410852] LNet: Added LNI 192.168.204.125@tcp34 [8/256/0/180] [19415.179802] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [19415.210458] LNet: Added LNI 192.168.204.125@tcp35 [8/256/0/180] [19422.910253] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [19422.953137] LNet: Added LNI 192.168.204.125@tcp36 [8/256/0/180] [19430.764187] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [19430.797783] LNet: Added LNI 192.168.204.125@tcp37 [8/256/0/180] [19438.319302] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [19438.362418] LNet: Added LNI 192.168.204.125@tcp38 [8/256/0/180] [19445.590511] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [19445.627499] LNet: Added LNI 192.168.204.125@tcp39 [8/256/0/180] [19452.973335] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [19453.003767] LNet: Added LNI 192.168.204.125@tcp40 [8/256/0/180] [19461.024974] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [19461.048396] LNet: Added LNI 192.168.204.125@tcp41 [8/256/0/180] [19468.403153] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [19468.433923] LNet: Added LNI 192.168.204.125@tcp42 [8/256/0/180] [19476.232214] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [19476.280151] LNet: Added LNI 192.168.204.125@tcp43 [8/256/0/180] [19483.946425] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [19483.971488] LNet: Added LNI 192.168.204.125@tcp44 [8/256/0/180] [19491.159303] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [19491.192816] LNet: Added LNI 192.168.204.125@tcp45 [8/256/0/180] [19498.397668] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [19498.428119] LNet: Added LNI 192.168.204.125@tcp46 [8/256/0/180] [19504.646624] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [19504.674220] LNet: Added LNI 192.168.204.125@tcp47 [8/256/0/180] [19512.644720] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [19512.678873] LNet: Added LNI 192.168.204.125@tcp48 [8/256/0/180] [19521.844854] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [19521.877948] LNet: Added LNI 192.168.204.125@tcp49 [8/256/0/180] [19531.982716] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [19532.029579] LNet: Added LNI 192.168.204.125@tcp50 [8/256/0/180] [19548.648274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [19548.657620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19548.671405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19552.741471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19552.742746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19552.770055] Lustre: Skipped 2 previous similar messages [19554.391537] Lustre: server umount lustre-MDT0000 complete [19557.926455] LustreError: 614823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171390 with bad export cookie 18066574687719319274 [19557.932880] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19557.937524] LustreError: 614823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [19558.219759] Lustre: server umount lustre-MDT0001 complete [19574.500753] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [19578.336317] Lustre: 614106:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788171395/real 1788171395] req@0000000075f4f910 x1875033207249664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788171411 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19578.360430] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19578.370617] Lustre: Skipped 1 previous similar message [19581.464169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19582.497331] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfab95df52992f0fa [19582.510204] Lustre: MGC192.168.204.125@tcp: Connection restored to 0@lo (at 0@lo) [19582.683325] LustreError: 616796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19582.698226] LustreError: 616796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [19582.730433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19583.392739] Lustre: 614103:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788171400/real 1788171400] req@000000005ea00dd0 x1875033207249920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788171416 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19583.776188] LustreError: 616797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19586.510913] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [19587.818080] LustreError: 616796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19589.856052] Lustre: 614103:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788171406/real 1788171406] req@00000000d45cf1ef x1875033207250048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788171422 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19594.212727] LustreError: 641038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19594.239964] LustreError: 641038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [19594.497425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19594.888284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19595.940703] Lustre: 614105:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788171411/real 1788171411] req@000000009afd70a9 x1875033207250176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788171427 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19598.827423] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [19600.363475] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [19605.662517] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19607.532051] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [19610.072779] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19628.012928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19628.023312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19628.032252] Lustre: Skipped 1 previous similar message [19628.048214] Lustre: Skipped 1 previous similar message [19636.193377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19636.199391] Lustre: Skipped 3 previous similar messages [19640.287316] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [19640.446168] Lustre: server umount lustre-OST0000 complete [19645.926418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [19645.943238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19645.965673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19649.675480] Lustre: server umount lustre-MDT0000 complete [19651.552923] LustreError: 641044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19651.592322] LustreError: 641044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [19653.218021] LustreError: 614824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171486 with bad export cookie 18066574687719321850 [19653.223647] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19653.239919] LustreError: 614824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [19653.703283] Lustre: server umount lustre-MDT0001 complete [19664.165505] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [19666.714610] Key type lgssc unregistered [19667.082566] LNet: 643822:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19667.090100] LNetError: 643822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19667.097264] LNet: Removed LNI 192.168.204.125@tcp [19667.103331] LNet: Removed LNI 192.168.204.125@tcp10 [19667.107672] LNet: Removed LNI 192.168.204.125@tcp11 [19667.112187] LNet: Removed LNI 192.168.204.125@tcp12 [19667.116600] LNet: Removed LNI 192.168.204.125@tcp13 [19667.120143] LNet: Removed LNI 192.168.204.125@tcp14 [19667.125809] LNet: Removed LNI 192.168.204.125@tcp15 [19667.127491] LNet: Removed LNI 192.168.204.125@tcp16 [19667.130934] LNet: Removed LNI 192.168.204.125@tcp17 [19667.134624] LNet: Removed LNI 192.168.204.125@tcp18 [19667.139819] LNet: Removed LNI 192.168.204.125@tcp19 [19667.141580] LNet: Removed LNI 192.168.204.125@tcp20 [19667.145364] LNet: Removed LNI 192.168.204.125@tcp21 [19667.148546] LNet: Removed LNI 192.168.204.125@tcp22 [19667.151767] LNet: Removed LNI 192.168.204.125@tcp23 [19667.155177] LNet: Removed LNI 192.168.204.125@tcp24 [19667.158607] LNet: Removed LNI 192.168.204.125@tcp25 [19667.160487] LNet: Removed LNI 192.168.204.125@tcp26 [19667.167963] LNet: Removed LNI 192.168.204.125@tcp27 [19667.178644] LNet: Removed LNI 192.168.204.125@tcp28 [19667.184376] LNet: Removed LNI 192.168.204.125@tcp29 [19667.190068] LNet: Removed LNI 192.168.204.125@tcp30 [19667.193960] LNet: Removed LNI 192.168.204.125@tcp31 [19667.198730] LNet: Removed LNI 192.168.204.125@tcp32 [19667.204391] LNet: Removed LNI 192.168.204.125@tcp33 [19667.208448] LNet: Removed LNI 192.168.204.125@tcp34 [19667.211589] LNet: Removed LNI 192.168.204.125@tcp35 [19667.215636] LNet: Removed LNI 192.168.204.125@tcp36 [19667.220899] LNet: Removed LNI 192.168.204.125@tcp37 [19667.226290] LNet: Removed LNI 192.168.204.125@tcp38 [19667.234297] LNet: Removed LNI 192.168.204.125@tcp39 [19667.237272] LNet: Removed LNI 192.168.204.125@tcp40 [19667.239522] LNet: Removed LNI 192.168.204.125@tcp41 [19667.244890] LNet: Removed LNI 192.168.204.125@tcp42 [19667.249926] LNet: Removed LNI 192.168.204.125@tcp43 [19667.253414] LNet: Removed LNI 192.168.204.125@tcp44 [19667.256049] LNet: Removed LNI 192.168.204.125@tcp45 [19667.260901] LNet: Removed LNI 192.168.204.125@tcp46 [19667.268019] LNet: Removed LNI 192.168.204.125@tcp47 [19667.272607] LNet: Removed LNI 192.168.204.125@tcp48 [19667.276377] LNet: Removed LNI 192.168.204.125@tcp49 [19667.291098] LNet: Removed LNI 192.168.204.125@tcp50 [19667.997870] Key type .llcrypt unregistered [19667.999932] Key type ._llcrypt unregistered [19687.817712] Key type ._llcrypt registered [19687.820290] Key type .llcrypt registered [19687.931150] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [19701.978306] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [19703.080264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19703.130115] alg: No test for adler32 (adler32-zlib) [19704.163736] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19704.425897] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [19706.135211] Key type lgssc registered [19707.395464] Lustre: Echo OBD driver; http://www.lustre.org/ [19740.275284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19740.300197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19741.540656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [19741.570720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [19741.634954] Lustre: lustre-MDT0000: new disk, initializing [19741.697192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19741.712109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19746.493751] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [19756.298988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19756.389732] Lustre: 647016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [19756.453915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [19756.460260] Lustre: Skipped 1 previous similar message [19756.528281] Lustre: lustre-MDT0001: new disk, initializing [19756.631891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19756.672203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [19756.684968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [19760.288635] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [19768.832153] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19774.333132] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19781.579199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19781.701629] Lustre: *** cfs_fail_loc=90f, val=0*** [19781.749890] Lustre: lustre-OST0000: new disk, initializing [19781.753841] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [19781.759702] Lustre: 649036:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19781.817805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19787.276598] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [19790.945037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19790.963750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19791.146904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19797.747249] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19806.694221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19806.694544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19806.715619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19806.732450] Lustre: Skipped 1 previous similar message [19811.499177] Lustre: server umount lustre-OST0000 complete [19816.938709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19816.942837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19816.950483] Lustre: Skipped 2 previous similar messages [19816.966158] Lustre: Skipped 1 previous similar message [19821.405574] Lustre: server umount lustre-MDT0000 complete [19824.559517] LustreError: 647009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171657 with bad export cookie 16651296526859089333 [19824.568237] LustreError: 647009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [19824.569842] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19824.796256] Lustre: server umount lustre-MDT0001 complete [19835.104464] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [19838.317088] Key type lgssc unregistered [19838.820111] LNet: 650940:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19838.825935] LNetError: 650940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19839.851358] LNet: Removed LNI 192.168.204.125@tcp [19840.727617] Key type .llcrypt unregistered [19840.730967] Key type ._llcrypt unregistered [19860.124460] Key type ._llcrypt registered [19860.125832] Key type .llcrypt registered [19860.201043] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [19861.075944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19861.153204] alg: No test for adler32 (adler32-zlib) [19862.187041] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19862.442517] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [19864.143157] Key type lgssc registered [19865.174703] Lustre: Echo OBD driver; http://www.lustre.org/ [19872.803786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19872.825839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19874.158641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19878.101251] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [19883.495188] LustreError: 652211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19883.511276] LustreError: 652211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [19886.095723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19886.439089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19890.440785] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [19896.997577] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19903.109139] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19910.068111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19910.406634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19916.558456] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [19923.724445] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19934.467353] loop0: detected capacity change from 0 to 204800000 [19934.615329] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19934.625916] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19934.638415] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19940.636537] loop0: detected capacity change from 0 to 204800000 [19940.820865] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19940.831934] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19940.841149] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19945.891619] loop0: detected capacity change from 0 to 204800000 [19951.188902] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19951.333573] Lustre: fs15246-OST0000: new disk, initializing [19951.337368] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [19951.348604] Lustre: 655806:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19951.428419] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19957.945422] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [19967.462258] loop1: detected capacity change from 0 to 204800000 [19973.194461] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19973.336895] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [19973.407280] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [19973.516278] Lustre: fs15246-MDT0000: new disk, initializing [19973.666757] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19973.689923] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19973.801879] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [19974.819428] Lustre: 656927:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000072efbb93 x1875033923303168/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788171823 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [19978.837415] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [19983.908432] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [19983.925415] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [19983.945948] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [20000.742396] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20000.761275] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [20005.861477] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [20006.657895] Lustre: server umount fs15246-MDT0000 complete [20010.988817] LustreError: 652206:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20015.330460] Lustre: server umount fs15246-OST0000 complete [20026.340116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20026.343986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [20026.349677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20026.360653] Lustre: Skipped 1 previous similar message [20031.463683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20031.472716] Lustre: Skipped 1 previous similar message [20036.582193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20036.594659] Lustre: Skipped 1 previous similar message [20040.160840] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [20040.427593] Lustre: server umount lustre-OST0000 complete [20045.283933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20045.290328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20045.301841] Lustre: Skipped 1 previous similar message [20045.308897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20050.508539] Lustre: server umount lustre-MDT0000 complete [20051.936995] LustreError: 652210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20051.955734] LustreError: 652210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [20054.496713] LustreError: 652190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171887 with bad export cookie 4492058560385948058 [20054.497932] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20054.505818] LustreError: 652190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [20054.851323] Lustre: server umount lustre-MDT0001 complete [20067.195459] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [20069.993412] Key type lgssc unregistered [20070.232253] LNet: 660025:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20070.243899] LNetError: 660025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20070.257094] LNet: Removed LNI 192.168.204.125@tcp [20071.017813] Key type .llcrypt unregistered [20071.020354] Key type ._llcrypt unregistered [20089.847941] Key type ._llcrypt registered [20089.850663] Key type .llcrypt registered [20089.990614] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [20091.423942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [20091.457609] alg: No test for adler32 (adler32-zlib) [20092.573422] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [20092.890747] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [20094.639287] Key type lgssc registered [20095.860473] Lustre: Echo OBD driver; http://www.lustre.org/ [20135.922000] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [20145.606229] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [20182.987937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20183.007884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20184.228716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [20184.256068] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [20184.334389] Lustre: lustre-MDT0000: new disk, initializing [20184.428421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20184.460431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20189.274235] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [20200.359839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20200.508280] Lustre: 664493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [20200.614036] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [20200.622316] Lustre: Skipped 1 previous similar message [20200.690065] Lustre: lustre-MDT0001: new disk, initializing [20200.789154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20200.822691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [20200.838234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [20205.600721] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [20213.911894] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20218.846430] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20225.009932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20225.252377] Lustre: lustre-OST0000: new disk, initializing [20225.256421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [20225.262234] Lustre: 666460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20225.347985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20231.853961] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [20233.342666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [20233.356934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [20233.469205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [20241.538513] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20245.761566] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [20245.984968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [20250.223909] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [20250.407131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [20254.179041] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20254.182656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [20254.199737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20254.220762] Lustre: Skipped 1 previous similar message [20258.427933] Lustre: server umount lustre-OST0000 complete [20262.374630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20262.385456] LustreError: Skipped 1 previous similar message [20262.387712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20262.406634] Lustre: Skipped 1 previous similar message [20262.418339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20264.416467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20264.424037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20264.435890] Lustre: Skipped 1 previous similar message [20267.891575] Lustre: server umount lustre-MDT0000 complete [20272.366097] LustreError: 667798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172105 with bad export cookie 888936449571112545 [20272.394610] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20272.863663] Lustre: server umount lustre-MDT0001 complete [20306.806488] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [20317.814625] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [20354.820331] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 20052 sec ========= 06:29:44 (1788172184) [20357.196728] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:29:46 (1788172186) === [20361.449430] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:29:50 (1788172190) === [20410.224837] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [20412.766460] Key type lgssc unregistered [20413.093445] LNet: 673109:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20413.100308] LNetError: 673109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20413.114141] LNet: Removed LNI 192.168.204.125@tcp [20413.878661] Key type .llcrypt unregistered [20413.882943] Key type ._llcrypt unregistered