[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 466377897 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002377] x2apic enabled [ 0.004004] Switched APIC routing to physical x2apic. [ 0.005009] kvm-guest: setup PV IPIs [ 0.008272] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010008] pid_max: default: 32768 minimum: 301 [ 0.011149] LSM: Security Framework initializing [ 0.012040] Yama: becoming mindful. [ 0.013024] SELinux: Initializing. [ 0.015009] *** VALIDATE selinux *** [ 0.022738] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027129] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028199] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029127] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030186] *** VALIDATE tmpfs *** [ 0.031653] *** VALIDATE proc *** [ 0.033258] *** VALIDATE cgroup *** [ 0.034041] *** VALIDATE cgroup2 *** [ 0.035490] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038028] Spectre V2 : User space: Vulnerable [ 0.039000] Speculative Store Bypass: Vulnerable [ 0.041063] debug: unmapping init [mem 0xffffffff98659000-0xffffffff98660fff] [ 0.043194] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044619] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045015] ... version: 2 [ 0.046008] ... bit width: 48 [ 0.047007] ... generic registers: 4 [ 0.048007] ... value mask: 0000ffffffffffff [ 0.049007] ... max period: 00007fffffffffff [ 0.050008] ... fixed-purpose events: 3 [ 0.051006] ... event mask: 000000070000000f [ 0.053226] rcu: Hierarchical SRCU implementation. [ 0.055575] smp: Bringing up secondary CPUs ... [ 0.056565] x86: Booting SMP configuration: [ 0.057014] .... node #0, CPUs: #1 #2 #3 [ 0.060605] smp: Brought up 1 node, 4 CPUs [ 0.062008] smpboot: Max logical packages: 1 [ 0.063015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143920] node 0 deferred pages initialised in 79ms [ 0.147117] devtmpfs: initialized [ 0.148335] x86/mm: Memory block size: 128MB [ 0.152087] gcov: version magic: 0x41383552 [ 0.154315] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158131] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160361] pinctrl core: initialized pinctrl subsystem [ 0.162161] [ 0.162511] ************************************************************* [ 0.164013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166008] ** ** [ 0.167008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170009] ** ** [ 0.171007] ** This means that this kernel is built to expose internal ** [ 0.173011] ** IOMMU data structures, which may compromise security on ** [ 0.175011] ** your system. ** [ 0.177010] ** ** [ 0.179009] ** If you see this message and you are not debugging the ** [ 0.182012] ** kernel, report this immediately to your vendor! ** [ 0.184009] ** ** [ 0.186013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187008] ************************************************************* [ 0.191000] NET: Registered protocol family 16 [ 0.192242] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.195059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.197055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.200179] cpuidle: using governor menu [ 0.203060] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205527] PCI: Using configuration type 1 for base access [ 0.208154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.219052] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.222049] cryptd: max_cpu_qlen set to 1000 [ 0.224198] ACPI: Added _OSI(Module Device) [ 0.225010] ACPI: Added _OSI(Processor Device) [ 0.226007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.228009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.231611] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.237605] ACPI: Interpreter enabled [ 0.239050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.240009] ACPI: Using IOAPIC for interrupt routing [ 0.242112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.244386] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.253228] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.255031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.257013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.260099] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.265264] acpiphp: Slot [2] registered [ 0.266105] acpiphp: Slot [3] registered [ 0.267074] acpiphp: Slot [4] registered [ 0.268064] acpiphp: Slot [5] registered [ 0.269102] acpiphp: Slot [6] registered [ 0.271103] acpiphp: Slot [7] registered [ 0.272102] acpiphp: Slot [8] registered [ 0.274133] acpiphp: Slot [9] registered [ 0.275066] acpiphp: Slot [10] registered [ 0.276073] acpiphp: Slot [11] registered [ 0.277088] acpiphp: Slot [12] registered [ 0.279064] acpiphp: Slot [13] registered [ 0.280074] acpiphp: Slot [14] registered [ 0.281060] acpiphp: Slot [15] registered [ 0.282066] acpiphp: Slot [16] registered [ 0.284073] acpiphp: Slot [17] registered [ 0.285059] acpiphp: Slot [18] registered [ 0.286057] acpiphp: Slot [19] registered [ 0.287063] acpiphp: Slot [20] registered [ 0.288090] acpiphp: Slot [21] registered [ 0.289067] acpiphp: Slot [22] registered [ 0.290088] acpiphp: Slot [23] registered [ 0.292062] acpiphp: Slot [24] registered [ 0.293071] acpiphp: Slot [25] registered [ 0.294071] acpiphp: Slot [26] registered [ 0.295108] acpiphp: Slot [27] registered [ 0.297074] acpiphp: Slot [28] registered [ 0.298057] acpiphp: Slot [29] registered [ 0.299075] acpiphp: Slot [30] registered [ 0.300156] acpiphp: Slot [31] registered [ 0.302084] PCI host bridge to bus 0000:00 [ 0.303013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.305014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.308015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.310012] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.312016] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.314017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.315205] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.319358] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.322438] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.331017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.333351] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.335008] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.337008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.338009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.341502] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.343600] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.345030] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.347618] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.350924] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.364022] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.367014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.372000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.378020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.386027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.410020] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.427000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.436017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.441019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.456028] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.465517] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.469031] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.476017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.493041] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.503908] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.508014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.514017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.528012] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.536486] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.544054] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.554055] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.569044] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.586510] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.595028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.602022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.617021] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.628419] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.630302] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.632318] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.634331] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.636177] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.641127] iommu: Default domain type: Passthrough [ 0.643761] SCSI subsystem initialized [ 0.646165] ACPI: bus type USB registered [ 0.647097] usbcore: registered new interface driver usbfs [ 0.651143] usbcore: registered new interface driver hub [ 0.653137] usbcore: registered new device driver usb [ 0.655187] pps_core: LinuxPPS API ver. 1 registered [ 0.657013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.660117] PTP clock support registered [ 0.662144] EDAC MC: Ver: 3.0.0 [ 0.663732] PCI: Using ACPI for IRQ routing [ 0.665121] NetLabel: Initializing [ 0.666010] NetLabel: domain hash size = 128 [ 0.668009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.671140] NetLabel: unlabeled traffic allowed by default [ 0.673416] vgaarb: loaded [ 0.675011] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.677012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.682818] clocksource: Switched to clocksource kvm-clock [ 0.798075] VFS: Disk quotas dquot_6.6.0 [ 0.799610] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.802215] *** VALIDATE ramfs *** [ 0.803460] *** VALIDATE hugetlbfs *** [ 0.805027] pnp: PnP ACPI init [ 0.807367] pnp: PnP ACPI: found 6 devices [ 0.824909] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.828576] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.830691] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.832773] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.835077] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.837470] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.840098] NET: Registered protocol family 2 [ 0.842415] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.847100] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.850890] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.855839] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.859280] TCP: Hash tables configured (established 65536 bind 65536) [ 0.862310] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.865425] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.868169] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.871288] NET: Registered protocol family 1 [ 0.874662] RPC: Registered named UNIX socket transport module. [ 0.877703] RPC: Registered udp transport module. [ 0.879599] RPC: Registered tcp transport module. [ 0.881417] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.884365] NET: Registered protocol family 44 [ 0.885796] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.887863] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.889969] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.892404] PCI: CLS 0 bytes, default 64 [ 0.893916] Unpacking initramfs... [ 2.536866] debug: unmapping init [mem 0xffff9d39fcc54000-0xffff9d39fffbffff] [ 2.542315] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.544793] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.548864] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.031145] Initialise system trusted keyrings [ 3.032796] Key type blacklist registered [ 3.036263] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.045304] zbud: loaded [ 3.049293] *** VALIDATE nfs *** [ 3.050481] *** VALIDATE nfs4 *** [ 3.051915] pstore: using deflate compression [ 3.055330] Platform Keyring initialized [ 3.253249] NET: Registered protocol family 38 [ 3.259119] Key type asymmetric registered [ 3.260640] Asymmetric key parser 'x509' registered [ 3.263137] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.267697] io scheduler mq-deadline registered [ 3.269441] io scheduler kyber registered [ 3.270873] io scheduler bfq registered [ 3.272457] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.276108] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.280804] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.284310] ACPI: Power Button [PWRF] [ 3.393575] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.496133] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.713968] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.843803] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.088334] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.125555] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.154139] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.158911] Non-volatile memory driver v1.3 [ 4.160505] Linux agpgart interface v0.103 [ 4.187698] virtio_blk virtio1: [vda] 133832 512-byte logical blocks (68.5 MB/65.3 MiB) [ 4.191531] vda: detected capacity change from 0 to 68521984 [ 4.209499] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.214374] vdb: detected capacity change from 0 to 1073741824 [ 4.234788] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.236697] vdc: detected capacity change from 0 to 2621440000 [ 4.253717] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.257529] vdd: detected capacity change from 0 to 2621440000 [ 4.277459] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.287187] vde: detected capacity change from 0 to 4294967296 [ 4.302725] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.306610] vdf: detected capacity change from 0 to 4294967296 [ 4.315650] libphy: Fixed MDIO Bus: probed [ 4.324988] usbcore: registered new interface driver usbserial_generic [ 4.327709] usbserial: USB Serial support registered for generic [ 4.329773] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.334916] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.336285] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.338610] mousedev: PS/2 mouse device common for all mice [ 4.341237] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.347209] rtc_cmos 00:05: RTC can wake from S4 [ 4.350845] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.355864] rtc_cmos 00:05: registered as rtc0 [ 4.357822] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.362793] intel_pstate: CPU model not supported [ 4.364494] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.380643] hid: raw HID events driver (C) Jiri Kosina [ 4.382516] usbcore: registered new interface driver usbhid [ 4.384450] usbhid: USB HID core driver [ 4.386139] drop_monitor: Initializing network drop monitor service [ 4.388973] Initializing XFRM netlink socket [ 4.390706] NET: Registered protocol family 10 [ 4.394238] Segment Routing with IPv6 [ 4.395660] NET: Registered protocol family 17 [ 4.399719] mpls_gso: MPLS GSO support [ 4.404297] RAS: Correctable Errors collector initialized. [ 4.406591] AVX version of gcm_enc/dec engaged. [ 4.408445] AES CTR mode by8 optimization enabled [ 4.542447] sched_clock: Marking stable (4542429923, 0)->(5411635468, -869205545) [ 4.545949] registered taskstats version 1 [ 4.552990] Loading compiled-in X.509 certificates [ 4.554669] zswap: loaded using pool lzo/zbud [ 4.592909] Key type big_key registered [ 4.612021] Key type encrypted registered [ 4.613245] ima: No TPM chip found, activating TPM-bypass! [ 4.616415] ima: Allocated hash algorithm: sha1 [ 4.618239] ima: No architecture policies found [ 4.624260] evm: Initialising EVM extended attributes: [ 4.625797] evm: security.selinux [ 4.626937] evm: security.ima [ 4.627878] evm: security.capability [ 4.634961] evm: HMAC attrs: 0x1 [ 4.638647] rtc_cmos 00:05: setting system clock to 2025-11-23 05:34:10 UTC (1763876050) [ 4.647437] debug: unmapping init [mem 0xffffffff99603000-0xffffffff997fffff] [ 4.650939] debug: unmapping init [mem 0xffffffff98382000-0xffffffff98658fff] [ 4.664139] Write protecting the kernel read-only data: 28672k [ 4.668137] debug: unmapping init [mem 0xffffffff96a03000-0xffffffff96bfffff] [ 4.671072] debug: unmapping init [mem 0xffffffff97314000-0xffffffff973fffff] [ 4.723359] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.731123] systemd[1]: Detected virtualization kvm. [ 4.732550] systemd[1]: Detected architecture x86-64. [ 4.734435] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.762870] systemd[1]: No hostname configured. [ 4.764294] systemd[1]: Set hostname to . [ 4.767199] random: systemd: uninitialized urandom read (16 bytes read) [ 4.770455] systemd[1]: Initializing machine ID from random generator. [ 4.838401] random: ln: uninitialized urandom read (6 bytes read) [ 4.986146] random: systemd: uninitialized urandom read (16 bytes read) [ 4.987888] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.992273] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.000629] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.917132] device-mapper: uevent: version 1.0.3 [ 5.920626] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.781430] virtio_net virtio0 ens2: renamed from eth0 [ 6.866975] random: fast init done [ 7.244235] scsi host0: ata_piix [ 7.306161] scsi host1: ata_piix [ 7.307485] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.317800] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.143439] random: crng init done [ 12.144889] random: 7 urandom warning(s) missed due to ratelimiting [ 13.649207] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 14.391619] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.797613] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.316604] SELinux: Disabled at runtime. [ 16.405117] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.419129] systemd[1]: Detected virtualization kvm. [ 16.421135] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.355997] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.363336] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.377301] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.385654] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.389911] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.411954] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.429920] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 17.648328] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 18.410732] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.845380] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.866864] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.083920] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.249544] EDAC sbridge: Ver: 1.1.2 [ 22.306700] Key type dns_resolver registered [ 22.617065] NFS: Registering the id_resolver key type [ 22.618934] Key type id_resolver registered [ 22.620278] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (7s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…-only root support (9s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit) [ *] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [*** ] A start job is running for Configur…only root support (13s / no limit) [** ] A start job is running for Configur…only root support (13s / no limit) [* ] A start job is running for Configur…only root support (14s / no limit) [** ] A start job is running for Configur…only root support (14s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg427-server login: [ 53.997582] libcfs: loading out-of-tree module taints kernel. [ 54.007221] Key type ._llcrypt registered [ 54.008222] Key type .llcrypt registered [ 54.046657] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 61.529662] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 62.107772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.115804] alg: No test for adler32 (adler32-zlib) [ 63.125287] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 63.416651] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 65.016186] Key type lgssc registered [ 65.528264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 71.801664] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 72.201348] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 75.094588] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.580250] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 80.104563] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.813922] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 88.724379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 88.745118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 88.751218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 89.839062] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 89.850307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 89.893209] Lustre: lustre-MDT0000: new disk, initializing [ 89.918742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 89.924808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 91.283729] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 96.544846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 96.569479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 96.596108] Lustre: 6487:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 96.606502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 96.608160] Lustre: Skipped 1 previous similar message [ 96.644182] Lustre: lustre-MDT0001: new disk, initializing [ 96.664141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 96.671976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 96.675103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 98.004435] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 100.333347] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 103.698321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 103.722993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 103.804293] Lustre: lustre-OST0000: new disk, initializing [ 103.805942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 103.822157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 104.363232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 104.365790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 104.387958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 105.655749] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.408594] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.432028] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 111.471595] Lustre: lustre-OST0001: new disk, initializing [ 111.474386] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 111.496278] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 112.372784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 112.376593] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 112.402219] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 113.712634] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 119.601705] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 123.285377] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 130.006239] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing check_logdir /tmp/testlogs/ [ 131.397935] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing yml_node [ 132.749394] Lustre: DEBUG MARKER: Client: 2.16.61.40 [ 133.494993] Lustre: DEBUG MARKER: MDS: 2.16.61.40 [ 134.255332] Lustre: DEBUG MARKER: OSS: 2.16.61.40 [ 134.759389] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 23 00:36:20 EST 2025 [ 139.986655] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 140.427502] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 153.056528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 153.060157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 153.065830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 153.568754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 153.569137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 153.573716] Lustre: Skipped 1 previous similar message [ 158.688770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 158.691598] Lustre: Skipped 5 previous similar messages [ 163.808719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 163.811085] Lustre: Skipped 3 previous similar messages [ 166.368124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 166.401456] LustreError: 12187:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 166.433447] Lustre: server umount lustre-MDT0000 complete [ 167.689889] LustreError: 6477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876213 with bad export cookie 263932928292677426 [ 167.691597] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 167.692522] LustreError: 6477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 168.928569] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 168.928836] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 168.933438] Lustre: Skipped 2 previous similar messages [ 173.818747] LustreError: 12390:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 173.821089] LustreError: 12390:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 173.856929] Lustre: server umount lustre-MDT0001 complete [ 181.228158] LustreError: 12591:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 181.230776] LustreError: 12591:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 181.254305] Lustre: server umount lustre-OST0000 complete [ 197.088227] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 197.118533] LustreError: 12792:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 197.121481] LustreError: 12792:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 197.208121] Lustre: server umount lustre-OST0001 complete [ 200.734326] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 205.549601] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 209.134912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 211.359433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 213.475495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 215.577809] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 217.738535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 217.762316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.855692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 217.869258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 217.902843] Lustre: lustre-MDT0000: new disk, initializing [ 217.927168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.933966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 219.223143] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 223.368022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 223.386551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 223.425925] Lustre: 14986:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 223.530945] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 224.867936] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 227.959525] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 229.126866] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 231.304085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 231.335932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 231.433889] Lustre: lustre-OST0000: new disk, initializing [ 231.435078] Lustre: Skipped 1 previous similar message [ 231.436610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 231.438199] Lustre: Skipped 2 previous similar messages [ 231.463942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 231.467686] Lustre: Skipped 1 previous similar message [ 232.567155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 232.569176] Lustre: Skipped 1 previous similar message [ 232.570509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 232.606239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 233.271308] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 236.450812] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 237.881920] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 237.955147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 239.299699] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 239.373781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 242.094754] LustreError: 17924:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 242.097191] LustreError: 17924:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 242.132595] Lustre: server umount lustre-OST0000 complete [ 244.193454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 244.196166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 244.201615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 244.204175] Lustre: Skipped 3 previous similar messages [ 248.289766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 253.414401] LustreError: 18125:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 253.416943] LustreError: 18125:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 253.443177] Lustre: server umount lustre-MDT0000 complete [ 254.746618] LustreError: 14979:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876300 with bad export cookie 263932928292678952 [ 254.749322] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 254.751977] LustreError: 14979:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 264.213848] Lustre: server umount lustre-MDT0001 complete [ 265.680904] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 00:38:31 (1763876311) [ 266.541674] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 267.107881] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 00:38:33 (1763876313) [ 269.308650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 269.462156] LustreError: 18997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 269.467660] LustreError: 18997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 269.485158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 270.789878] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 273.868228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 275.429782] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 277.264208] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 278.476039] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 280.718122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 282.695649] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 284.461272] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 304.611135] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 304.614964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.626988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 304.629103] Lustre: Skipped 4 previous similar messages [ 309.168204] LustreError: 21263:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 309.170565] LustreError: 21263:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 309.219751] Lustre: server umount lustre-OST0000 complete [ 316.823898] Lustre: server umount lustre-MDT0000 complete [ 318.130576] LustreError: 18976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876363 with bad export cookie 263932928292679834 [ 318.133366] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 318.136421] LustreError: 18976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 318.257607] Lustre: server umount lustre-MDT0001 complete [ 321.786674] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 322.802327] Key type lgssc unregistered [ 322.934325] LNet: 22198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.936703] LNetError: 22198:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 322.946354] LNet: Removed LNI 192.168.204.127@tcp [ 323.280128] Key type .llcrypt unregistered [ 323.281393] Key type ._llcrypt unregistered [ 331.952155] Key type ._llcrypt registered [ 331.953134] Key type .llcrypt registered [ 331.988092] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 332.296182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 332.306569] alg: No test for adler32 (adler32-zlib) [ 333.166611] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 333.256489] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 334.840167] Key type lgssc registered [ 335.201543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 337.941675] loop: module loaded [ 338.086344] loop0: detected capacity change from 0 to 204800000 [ 338.148885] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 338.154735] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 338.267362] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 338.291053] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 340.238535] loop0: detected capacity change from 0 to 204800000 [ 340.309866] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 340.313857] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 340.318147] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 340.367942] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 342.065790] loop0: detected capacity change from 0 to 204800000 [ 344.103735] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 344.121862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 344.127147] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 345.218665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 345.237193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 345.274401] Lustre: lustre-MDT0000: new disk, initializing [ 345.298186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 345.306174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 346.726362] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 350.550818] loop1: detected capacity change from 0 to 204800000 [ 352.804033] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 352.839756] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 352.956927] Lustre: lustre-OST07c6: new disk, initializing [ 352.959890] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 352.962840] Lustre: Skipped 1 previous similar message [ 352.994554] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 355.172239] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 360.454519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 360.458386] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 360.470619] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 362.781346] Lustre: 25344:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 362.785064] Lustre: 25344:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 362.787741] Lustre: 25344:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 362.790765] Lustre: 25344:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 362.793791] Lustre: 25344:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 362.796887] Lustre: 25344:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 377.020802] Lustre: 24339:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 377.023160] Lustre: 24339:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 377.026455] Lustre: 24339:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 377.028145] Lustre: 24339:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 377.030869] Lustre: 24339:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 377.033693] Lustre: 24339:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 380.560753] Lustre: 24339:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 380.564909] Lustre: 24339:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 380.568315] Lustre: 24339:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 380.570500] Lustre: 24339:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 380.572734] Lustre: 24339:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 380.575185] Lustre: 24339:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 380.577702] Lustre: 24339:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 380.580497] Lustre: 24339:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 380.582456] Lustre: 24339:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 380.584399] Lustre: 24339:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 380.586233] Lustre: 24339:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 380.588607] Lustre: 24339:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 383.646499] Lustre: 24339:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 383.650939] Lustre: 24339:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 383.654554] Lustre: 24339:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 383.658194] Lustre: 24339:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 383.661814] Lustre: 24339:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 383.665605] Lustre: 24339:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 383.669277] Lustre: 24339:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 383.673340] Lustre: 24339:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 383.676919] Lustre: 24339:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 383.680105] Lustre: 24339:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 383.683584] Lustre: 24339:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 383.687045] Lustre: 24339:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 390.245702] Lustre: 24338:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 390.247952] Lustre: 24338:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 3 previous similar messages [ 390.251559] Lustre: 24338:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 390.253391] Lustre: 24338:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 390.255122] Lustre: 24338:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 390.257975] Lustre: 24338:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 390.259970] Lustre: 24338:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 390.262335] Lustre: 24338:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 390.265444] Lustre: 24338:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 390.267392] Lustre: 24338:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 390.269196] Lustre: 24338:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 390.271235] Lustre: 24338:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 400.242881] Lustre: 24339:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 400.246215] Lustre: 24339:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 5 previous similar messages [ 400.248462] Lustre: 24339:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 400.250884] Lustre: 24339:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 400.253103] Lustre: 24339:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 400.255430] Lustre: 24339:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 400.258236] Lustre: 24339:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 400.261157] Lustre: 24339:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 400.263404] Lustre: 24339:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 400.265778] Lustre: 24339:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 400.268696] Lustre: 24339:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 400.270871] Lustre: 24339:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 442.338220] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 442.338541] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 442.342066] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 447.457374] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 447.532991] LustreError: 26545:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 447.567847] Lustre: server umount lustre-OST07c6 complete [ 450.905229] LustreError: 27046:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 450.907379] LustreError: 27046:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 450.963770] Lustre: server umount lustre-MDT0000 complete [ 456.534497] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 457.630453] Key type lgssc unregistered [ 457.761509] LNet: 27878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 457.764308] LNetError: 27878:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 457.774517] LNet: Removed LNI 192.168.204.127@tcp [ 458.141213] Key type .llcrypt unregistered [ 458.142241] Key type ._llcrypt unregistered [ 465.747282] Key type ._llcrypt registered [ 465.748431] Key type .llcrypt registered [ 465.785236] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 466.214989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 466.273116] alg: No test for adler32 (adler32-zlib) [ 467.157698] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 467.252484] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 468.840214] Key type lgssc registered [ 469.273604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 472.088050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 472.095361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.241606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 474.538557] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 477.421848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 477.541313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 478.823301] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 480.532397] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 481.710833] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 483.904231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 484.017044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.775086] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 487.397787] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 498.341667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 508.897598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.899491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 508.901591] Lustre: Skipped 1 previous similar message [ 508.901971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 513.327113] LustreError: 31578:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 513.364575] Lustre: server umount lustre-OST0000 complete [ 518.624855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 518.627146] LustreError: Skipped 1 previous similar message [ 518.628427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.631881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 518.633295] Lustre: Skipped 1 previous similar message [ 520.828194] LustreError: 31781:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 520.830919] LustreError: 31781:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 520.859223] Lustre: server umount lustre-MDT0000 complete [ 522.017231] LustreError: 29087:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876567 with bad export cookie 11342785117197220999 [ 522.019743] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 522.021152] LustreError: 29087:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 522.088984] LustreError: 31982:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 522.091904] LustreError: 31982:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 522.139030] Lustre: server umount lustre-MDT0001 complete [ 525.507899] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 526.552088] Key type lgssc unregistered [ 526.688387] LNet: 32513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 526.691944] LNetError: 32513:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 526.699341] LNet: Removed LNI 192.168.204.127@tcp [ 527.008737] Key type .llcrypt unregistered [ 527.009882] Key type ._llcrypt unregistered [ 533.761076] Key type ._llcrypt registered [ 533.762686] Key type .llcrypt registered [ 533.796139] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 534.096814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 534.112889] alg: No test for adler32 (adler32-zlib) [ 534.961854] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 535.040450] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 536.624156] Key type lgssc registered [ 536.987677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 539.535105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 539.541961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.694686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 541.864327] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 544.591449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 544.705107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 545.911521] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 547.591243] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 548.680269] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 550.832458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 550.884066] LustreError: 35352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.889809] LustreError: 35352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 550.935105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 552.706444] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 554.390380] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 555.051826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 565.218669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 565.221019] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 565.221257] LustreError: Skipped 1 previous similar message [ 565.225586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 565.227706] Lustre: Skipped 1 previous similar message [ 570.287119] LustreError: 36000:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 570.323718] Lustre: server umount lustre-OST0000 complete [ 575.457418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 575.457422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 575.457430] Lustre: Skipped 1 previous similar message [ 575.458144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 577.596419] LustreError: 36202:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 577.598290] LustreError: 36202:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 577.622576] Lustre: server umount lustre-MDT0000 complete [ 578.830946] LustreError: 33734:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876624 with bad export cookie 2021552983516725319 [ 578.833677] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.834910] LustreError: 33734:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 578.898607] LustreError: 36403:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 578.901043] LustreError: 36403:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 578.950348] Lustre: server umount lustre-MDT0001 complete [ 582.402448] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 583.350186] Key type lgssc unregistered [ 583.468395] LNet: 36934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 583.470957] LNetError: 36934:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 583.477290] LNet: Removed LNI 192.168.204.127@tcp [ 583.763437] Key type .llcrypt unregistered [ 583.764385] Key type ._llcrypt unregistered [ 592.179597] Key type ._llcrypt registered [ 592.181300] Key type .llcrypt registered [ 592.217648] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 592.511920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 592.519375] alg: No test for adler32 (adler32-zlib) [ 593.364638] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 593.440783] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 595.016143] Key type lgssc registered [ 595.341030] Lustre: Echo OBD driver; http://www.lustre.org/ [ 597.752419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 597.759469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 598.901781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 600.077068] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 602.653297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 602.764868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 603.977960] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 605.693719] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 606.886316] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 609.034388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 609.134350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 610.150180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 610.921736] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 612.649127] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 615.867842] LustreError: 40555:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 615.894512] Lustre: server umount lustre-MDT0000 complete [ 617.046255] LustreError: 38290:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876662 with bad export cookie 3287649066880725987 [ 617.050034] LustreError: 38290:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 617.050142] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 617.441146] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 617.441666] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 617.446035] Lustre: Skipped 1 previous similar message [ 617.450227] LustreError: 40757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.454235] LustreError: 40757:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 619.134248] LustreError: 40755:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 619.136012] LustreError: 40755:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 619.179905] Lustre: server umount lustre-MDT0001 complete [ 630.049549] LustreError: 38290:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.27@tcp arrived at 1763876675 with bad export cookie 3287649066880726834 [ 630.054223] LustreError: 38290:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 636.973502] LustreError: 40979:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 636.975238] LustreError: 40979:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 637.012667] Lustre: server umount lustre-OST0000 complete [ 641.979890] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 642.908277] Key type lgssc unregistered [ 643.029357] LNet: 41815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.032171] LNetError: 41815:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 643.037248] LNet: Removed LNI 192.168.204.127@tcp [ 643.296889] Key type .llcrypt unregistered [ 643.298590] Key type ._llcrypt unregistered [ 651.735502] Key type ._llcrypt registered [ 651.736810] Key type .llcrypt registered [ 651.776898] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 652.103895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 652.111165] alg: No test for adler32 (adler32-zlib) [ 652.967485] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 653.051682] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 654.640135] Key type lgssc registered [ 655.048139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 657.676540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 657.684691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 658.859771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 660.273242] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 663.307575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 663.433944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 664.821437] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 666.778829] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 668.121419] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 670.516621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 670.636483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 672.625766] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 674.533592] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 683.943567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 689.124247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 689.124669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.126049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 689.127232] LustreError: Skipped 1 previous similar message [ 689.133769] Lustre: Skipped 1 previous similar message [ 690.404853] Lustre: lustre-OST0000: Not available for connect from 192.168.204.27@tcp (stopping) [ 690.407410] Lustre: Skipped 1 previous similar message [ 693.616108] LustreError: 45458:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 693.651701] Lustre: server umount lustre-OST0000 complete [ 706.451817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 706.457502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 706.459294] Lustre: Skipped 1 previous similar message [ 709.537190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.539406] Lustre: Skipped 1 previous similar message [ 712.641054] LustreError: 45833:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 712.643425] LustreError: 45833:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 712.676488] Lustre: server umount lustre-MDT0000 complete [ 713.941384] LustreError: 43173:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876759 with bad export cookie 5733592968614441617 [ 713.944675] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 713.945632] LustreError: 43173:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 720.064860] LustreError: 46034:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 720.067982] LustreError: 46034:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 720.122960] Lustre: server umount lustre-MDT0001 complete [ 723.704771] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 724.740298] Key type lgssc unregistered [ 724.876399] LNet: 46567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.880434] LNetError: 46567:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 724.890328] LNet: Removed LNI 192.168.204.127@tcp [ 725.211635] Key type .llcrypt unregistered [ 725.213647] Key type ._llcrypt unregistered [ 734.217480] Key type ._llcrypt registered [ 734.218607] Key type .llcrypt registered [ 734.255096] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 734.654972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 734.685218] alg: No test for adler32 (adler32-zlib) [ 735.542050] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 735.630257] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 737.216139] Key type lgssc registered [ 737.577872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 740.131920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 740.139199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 741.293581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 742.608376] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 745.472866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 745.593131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 746.995791] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 748.838538] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 750.107773] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 752.360909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 752.484166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 754.479367] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 756.270294] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 766.826950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 773.828891] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 774.964128] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 785.658133] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-11-23 0h47m11s [ 786.173788] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 786.912781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 786.915856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 786.920303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 787.937971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 787.940683] Lustre: Skipped 5 previous similar messages [ 794.084519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 794.086314] Lustre: Skipped 3 previous similar messages [ 799.202434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 799.204859] Lustre: Skipped 1 previous similar message [ 801.248140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 801.281609] LustreError: 50524:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 801.309235] Lustre: server umount lustre-MDT0000 complete [ 802.581038] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 804.891702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 804.931406] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 805.018894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 805.041816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 806.056202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 806.382923] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 807.539255] Lustre: DEBUG MARKER: Wait for df (35912) ... [ 808.113598] Lustre: DEBUG MARKER: done [ 810.465433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 810.468226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 810.472010] Lustre: Skipped 2 previous similar messages [ 810.474065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 810.476367] Lustre: Skipped 4 previous similar messages [ 811.247202] LustreError: 51576:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 811.250406] LustreError: 51576:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 811.288728] Lustre: server umount lustre-OST0000 complete [ 814.653344] LustreError: 51779:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 814.655613] LustreError: 51779:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 814.683123] Lustre: server umount lustre-MDT0000 complete [ 815.930269] LustreError: 47920:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876861 with bad export cookie 14064729108164872788 [ 815.935874] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 825.315817] LustreError: 51980:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 825.317784] LustreError: 51980:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 825.361730] Lustre: server umount lustre-MDT0001 complete [ 828.847503] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 829.815815] Key type lgssc unregistered [ 829.943346] LNet: 52512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.946225] LNetError: 52512:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.955345] LNet: Removed LNI 192.168.204.127@tcp [ 830.251701] Key type .llcrypt unregistered [ 830.252743] Key type ._llcrypt unregistered [ 841.388838] Key type ._llcrypt registered [ 841.390166] Key type .llcrypt registered [ 841.431978] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 847.705206] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 848.032814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 848.040046] alg: No test for adler32 (adler32-zlib) [ 848.898180] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 848.985754] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 850.568127] Key type lgssc registered [ 850.925042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 853.746072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 855.882290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 857.920389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 860.176988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 862.494176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 862.516110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 862.521509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 863.629711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 863.643687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 863.685702] Lustre: lustre-MDT0000: new disk, initializing [ 863.711616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.718139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 865.076677] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 869.331213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 869.357983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 869.395123] Lustre: 56433:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 869.420379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 869.422023] Lustre: Skipped 1 previous similar message [ 869.462362] Lustre: lustre-MDT0001: new disk, initializing [ 869.498606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 869.513834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 869.518298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 870.893759] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 874.100662] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 875.266632] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 877.547737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 877.574308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 877.672353] Lustre: lustre-OST0000: new disk, initializing [ 877.674461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 877.708378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 879.622338] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 882.685243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 882.689130] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 882.723277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 882.798745] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 884.234647] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 884.309973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 885.737265] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 885.833905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 887.781642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.782379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 887.788086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 887.789663] LustreError: Skipped 1 previous similar message [ 892.591851] LustreError: 59370:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 892.631763] Lustre: server umount lustre-OST0000 complete [ 895.456921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 895.460361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 895.464291] Lustre: Skipped 1 previous similar message [ 895.466146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 895.468193] Lustre: Skipped 1 previous similar message [ 898.017157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 898.017700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 903.137093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 903.139156] Lustre: Skipped 2 previous similar messages [ 907.233318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 907.235144] Lustre: Skipped 1 previous similar message [ 908.256154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 908.293969] LustreError: 59571:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 908.297166] LustreError: 59571:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 908.329939] Lustre: server umount lustre-MDT0000 complete [ 909.585098] LustreError: 56424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876955 with bad export cookie 15601575316135874349 [ 909.588114] LustreError: 56424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 909.589672] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 909.661487] LustreError: 59776:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 909.663255] LustreError: 59776:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 909.718273] Lustre: server umount lustre-MDT0001 complete [ 912.445130] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 00:49:18 (1763876958) [ 914.647091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 914.809545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 916.081473] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 918.939523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 919.059627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 920.391044] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 922.190532] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 923.490313] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 925.729769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 927.823640] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 929.681125] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 942.572831] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-11-23 0h49m48s [ 943.076851] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 945.419786] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 946.270953] Lustre: DEBUG MARKER: Creating a test file and stat it [ 946.284285] LustreError: 61768:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 966.383501] Lustre: lustre-MDT0000: Client 50195976-fe90-4fda-a0a0-4940e7ccafe4 (at 192.168.204.27@tcp) reconnecting [ 986.328077] LustreError: 61768:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 987.039902] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 987.835167] Lustre: DEBUG MARKER: done [ 990.689896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 990.692298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 990.697242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 995.502102] LustreError: 63470:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 995.504760] LustreError: 63470:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 995.544347] Lustre: server umount lustre-OST0000 complete [ 996.833277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 996.836532] Lustre: Skipped 2 previous similar messages [ 1009.121883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1009.123433] Lustre: Skipped 9 previous similar messages [ 1011.168163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1011.213758] LustreError: 63672:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1011.215461] LustreError: 63672:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1011.244770] Lustre: server umount lustre-MDT0000 complete [ 1012.841720] LustreError: 60425:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877058 with bad export cookie 15601575316135875217 [ 1012.845164] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1012.847261] LustreError: 60425:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1012.993551] Lustre: server umount lustre-MDT0001 complete [ 1016.900193] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1017.978750] Key type lgssc unregistered [ 1018.116377] LNet: 64406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.118991] LNetError: 64406:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.126325] LNet: Removed LNI 192.168.204.127@tcp [ 1018.421491] Key type .llcrypt unregistered [ 1018.422530] Key type ._llcrypt unregistered [ 1029.374673] Key type ._llcrypt registered [ 1029.376655] Key type .llcrypt registered [ 1029.415592] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 1035.589820] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1035.906914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1035.950113] alg: No test for adler32 (adler32-zlib) [ 1036.832398] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1036.925728] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1038.504202] Key type lgssc registered [ 1038.942292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1041.857036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1043.903753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1045.934032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1048.046684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1050.323459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1050.344649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1050.350175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.440832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1051.453304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1051.492268] Lustre: lustre-MDT0000: new disk, initializing [ 1051.516878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.523773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1052.822170] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1056.910301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1056.940191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1056.974554] Lustre: 68328:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1057.004152] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1057.006020] Lustre: Skipped 1 previous similar message [ 1057.040591] Lustre: lustre-MDT0001: new disk, initializing [ 1057.080983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1057.095540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1057.099459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1058.464698] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1061.539875] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1062.736675] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1065.034693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1065.061814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1065.155043] Lustre: lustre-OST0000: new disk, initializing [ 1065.157177] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1065.182561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.174199] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1068.027803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1068.031553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1068.065681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1070.249567] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1071.651538] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1071.736017] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1073.049396] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1073.130231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1078.244678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1078.245446] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1078.252294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1078.253723] LustreError: Skipped 1 previous similar message [ 1079.854824] LustreError: 71265:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1079.894500] Lustre: server umount lustre-OST0000 complete [ 1082.848902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1082.850940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.854603] Lustre: Skipped 1 previous similar message [ 1082.856426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1082.858717] Lustre: Skipped 1 previous similar message [ 1087.229310] LustreError: 71466:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1087.231176] LustreError: 71466:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1087.257859] Lustre: server umount lustre-MDT0000 complete [ 1088.471213] LustreError: 70072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877134 with bad export cookie 4341955474321573776 [ 1088.474201] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1088.475034] LustreError: 70072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1088.539064] LustreError: 71668:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1088.541072] LustreError: 71668:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1088.588071] Lustre: server umount lustre-MDT0001 complete [ 1091.497273] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 00:52:17 (1763877137) [ 1092.022345] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1092.636164] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 00:52:18 (1763877138) [ 1096.890774] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1098.310071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1098.454202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1098.773655] Lustre: Failing over lustre-MDT0000 [ 1098.775324] LustreError: 72742:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1098.778849] LustreError: 72742:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1098.782056] LustreError: 72742:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1098.841698] LustreError: 72793:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1098.843285] LustreError: 72793:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1098.884590] Lustre: server umount lustre-MDT0000 complete [ 1101.030465] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 00:52:27 (1763877147) [ 1103.337841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.496594] LustreError: 73360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1103.503094] LustreError: 73360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1103.513698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1104.817092] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1107.771404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1109.298826] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1111.144996] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1112.403166] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1114.774970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1114.891533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.893522] Lustre: Skipped 1 previous similar message [ 1116.822166] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1118.588708] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1131.313479] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1133.539614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1133.540472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1133.541787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1133.541797] Lustre: Skipped 2 previous similar messages [ 1133.542856] Lustre: Skipped 3 previous similar messages [ 1138.659691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1138.661266] Lustre: Skipped 3 previous similar messages [ 1139.197346] LustreError: 75723:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1139.199362] LustreError: 75723:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1139.225887] Lustre: server umount lustre-MDT0000 complete [ 1140.427322] LustreError: 73342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877186 with bad export cookie 4341955474321574917 [ 1140.431458] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1140.431651] LustreError: 73342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1140.539334] Lustre: server umount lustre-MDT0001 complete [ 1141.726142] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1143.263625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1146.258577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1150.944633] LustreError: 66541:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000051807132 x1849559175193472/t0(0) o250->MGC192.168.204.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1151.051787] LustreError: 74967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1151.052628] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.085172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1151.108300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1152.357845] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1155.740051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1157.332741] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1159.200123] Lustre: 66542:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763877189/real 1763877189] req@00000000d7e48d5c x1849559175193088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763877205 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1159.421100] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1160.747929] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1161.198136] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1162.208246] Lustre: 66543:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763877191/real 1763877191] req@000000004eb8ab53 x1849559175193216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763877207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1163.241637] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1164.256112] Lustre: 66544:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763877194/real 1763877194] req@00000000825c8c8a x1849559175193344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763877210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1166.304270] Lustre: 66545:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763877196/real 1763877196] req@000000001f8bb316 x1849559175193600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763877212 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1168.357066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1168.358537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1168.360876] Lustre: Skipped 2 previous similar messages [ 1168.362834] Lustre: Skipped 1 previous similar message [ 1173.886281] LustreError: 78184:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1173.888534] LustreError: 78184:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1173.914964] Lustre: server umount lustre-MDT0000 complete [ 1175.217361] LustreError: 73342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877221 with bad export cookie 4341955474321576807 [ 1175.220636] LustreError: 73342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1175.221252] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1175.326464] Lustre: server umount lustre-MDT0001 complete [ 1177.554896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1181.131561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1185.761378] LustreError: 66541:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000031a1e91c x1849559175214080/t0(0) o250->MGC192.168.204.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1185.920581] LustreError: 74967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1185.929266] LustreError: 74967:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1185.950564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1185.955083] Lustre: Skipped 1 previous similar message [ 1185.982972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1187.272363] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1190.199972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1190.286719] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.579369] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1193.364893] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1194.400126] Lustre: 66544:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763877224/real 1763877224] req@00000000f2968e97 x1849559175213696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763877240 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1194.535794] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1195.494360] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1198.566359] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1203.688612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1203.692107] Lustre: Skipped 6 previous similar messages [ 1207.497133] LustreError: 80637:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1207.500738] LustreError: 80637:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1207.533709] Lustre: server umount lustre-MDT0000 complete [ 1208.803439] LustreError: 79238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1208.812478] LustreError: 79238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1209.185372] LustreError: 75926:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877255 with bad export cookie 4341955474321578179 [ 1209.189630] LustreError: 75926:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1209.190300] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1209.297367] Lustre: server umount lustre-MDT0001 complete [ 1211.466525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1212.540827] Lustre: DEBUG MARKER: files compared the same [ 1222.419628] Lustre: server umount lustre-OST0000 complete [ 1228.086367] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1229.096355] Key type lgssc unregistered [ 1229.229475] LNet: 82268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1229.231677] LNetError: 82268:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1229.237573] LNet: Removed LNI 192.168.204.127@tcp [ 1229.569058] Key type .llcrypt unregistered [ 1229.570200] Key type ._llcrypt unregistered [ 1238.003328] Key type ._llcrypt registered [ 1238.005283] Key type .llcrypt registered [ 1238.047203] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1238.377611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1238.418267] alg: No test for adler32 (adler32-zlib) [ 1239.265043] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1239.342414] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1240.920216] Key type lgssc registered [ 1241.282056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1243.824604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1243.830594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1244.986304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1246.268333] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug +malloc all [ 1249.021561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1249.114186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1250.358669] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug +malloc all [ 1252.187891] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1253.337088] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1255.498489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1255.584518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1257.358532] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug +malloc all [ 1259.112018] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1269.921981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1279.456654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1279.459025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1279.462879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1280.481083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1280.481446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1280.481636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1283.118295] LustreError: 85911:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1283.150573] Lustre: server umount lustre-OST0000 complete [ 1285.088615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1285.090681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.093841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1285.095406] Lustre: Skipped 1 previous similar message [ 1290.492269] LustreError: 86113:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1290.494207] LustreError: 86113:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1290.513987] Lustre: server umount lustre-MDT0000 complete [ 1290.720528] LustreError: 84367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1290.726280] LustreError: 84367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1291.758875] LustreError: 83623:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877337 with bad export cookie 14108470785532439521 [ 1291.760765] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1291.764570] LustreError: 83623:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1291.829191] LustreError: 86315:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1291.830930] LustreError: 86315:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1291.875548] Lustre: server umount lustre-MDT0001 complete [ 1295.877634] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1296.914248] Key type lgssc unregistered [ 1297.050380] LNet: 86847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1297.052530] LNetError: 86847:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1297.061456] LNet: Removed LNI 192.168.204.127@tcp [ 1297.390628] Key type .llcrypt unregistered [ 1297.392593] Key type ._llcrypt unregistered [ 1306.156741] Key type ._llcrypt registered [ 1306.158073] Key type .llcrypt registered [ 1306.197158] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1306.559189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1306.573954] alg: No test for adler32 (adler32-zlib) [ 1307.442810] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1307.540815] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1309.128192] Key type lgssc registered [ 1309.617759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1312.187658] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1312.193263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1327.648310] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1332.768168] LustreError: 88194:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.127@tcp: failed processing log, type 4: rc = -110 [ 1359.392184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.124268] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1362.902771] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1365.434954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1365.468693] Lustre: *** cfs_fail_loc=706, val=0*** [ 1365.601749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1365.623110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1366.897278] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1369.632665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1369.768532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1371.064097] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1372.876702] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1374.090789] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1375.204400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.204532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1375.205299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1375.207679] Lustre: Skipped 1 previous similar message [ 1376.814115] LustreError: 90511:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1376.847493] Lustre: server umount lustre-OST0000 complete [ 1380.321795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1380.322262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1380.329909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1380.331307] Lustre: Skipped 2 previous similar messages [ 1384.132537] LustreError: 90713:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1384.135889] LustreError: 90713:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1384.168878] Lustre: server umount lustre-MDT0000 complete [ 1385.408322] LustreError: 90715:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877431 with bad export cookie 11266134467898945551 [ 1385.411607] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1385.413286] LustreError: 90715:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1385.481040] LustreError: 90916:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1385.482698] LustreError: 90916:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1385.537978] Lustre: server umount lustre-MDT0001 complete [ 1389.064386] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1390.066349] Key type lgssc unregistered [ 1390.196471] LNet: 91447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1390.198592] LNetError: 91447:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.207412] LNet: Removed LNI 192.168.204.127@tcp [ 1390.506227] Key type .llcrypt unregistered [ 1390.507259] Key type ._llcrypt unregistered [ 1399.850410] Key type ._llcrypt registered [ 1399.851314] Key type .llcrypt registered [ 1399.896656] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1400.287393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1400.331042] alg: No test for adler32 (adler32-zlib) [ 1401.195555] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1401.282820] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1402.872106] Key type lgssc registered [ 1403.235483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1405.975217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1405.982190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1408.277746] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1410.906150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1411.069389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1412.336449] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1415.182995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1415.305381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1417.158933] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1419.951107] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1419.954918] Lustre: MGS: Client 68fb334e-02fd-472f-90c1-b05107c19bdb (at 0@lo) reconnecting [ 1420.038183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1420.063859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1421.340268] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1430.497159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1430.499146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1430.502477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1431.522152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1431.523899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1431.524419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.182096] LustreError: 95455:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1435.218163] Lustre: server umount lustre-OST0000 complete [ 1436.641554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1436.641736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1436.642395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1436.642401] Lustre: Skipped 1 previous similar message [ 1436.652929] Lustre: Skipped 1 previous similar message [ 1441.761862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1441.763906] Lustre: Skipped 2 previous similar messages [ 1442.627870] LustreError: 95657:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1442.629565] LustreError: 95657:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1442.638957] Lustre: server umount lustre-MDT0000 complete [ 1444.130562] LustreError: 95864:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1444.133927] LustreError: 95864:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1444.179516] Lustre: server umount lustre-MDT0001 complete [ 1445.515820] Lustre: server umount MGS complete [ 1449.131284] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 00:58:15 (1763877495) [ 1458.416606] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 1461.211761] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1464.253378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1466.384595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1468.388597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1470.329251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1472.529372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1472.550887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.895104] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1477.688197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1477.708684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1477.783611] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1477.801517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1477.839311] Lustre: lustre-MDT0001: new disk, initializing [ 1477.862869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1477.870485] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1477.872628] Lustre: 100572:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1477.875153] LustreError: 100572:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1479.119327] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1483.191113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1483.211704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1483.300123] Lustre: lustre-OST0000: new disk, initializing [ 1483.301806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1483.321801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1485.097994] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1486.816685] Lustre: 101763:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000005243bc95 x1849559557899008/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763877548 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1488.033530] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1490.175247] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1490.179277] Lustre: MGS: Client c95eadb3-7cb5-452e-8786-06ca48a97952 (at 0@lo) reconnecting [ 1490.194535] Lustre: 99857:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1490.201830] LustreError: 99857:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1490.211113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1490.250600] Lustre: lustre-MDT0000: new disk, initializing [ 1490.304126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1491.363969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1491.367780] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1491.376874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1491.522830] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1510.881195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1510.883535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1510.887401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1516.143114] LustreError: 102805:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1516.145016] LustreError: 102805:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1516.180825] Lustre: server umount lustre-OST0000 complete [ 1522.145942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1522.146028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.146565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1522.146569] Lustre: Skipped 2 previous similar messages [ 1522.154871] Lustre: Skipped 2 previous similar messages [ 1523.710992] LustreError: 103008:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1523.713366] LustreError: 103008:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1523.723802] Lustre: server umount lustre-MDT0000 complete [ 1531.119625] Lustre: server umount lustre-MDT0001 complete [ 1532.312069] LustreError: 103410:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1532.313913] LustreError: 103410:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1532.333809] Lustre: server umount MGS complete [ 1536.149429] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 00:59:42 (1763877582) [ 1541.860050] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1542.863684] Key type lgssc unregistered [ 1542.985425] LNet: 104744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1542.987728] LNetError: 104744:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1542.994297] LNet: Removed LNI 192.168.204.127@tcp [ 1543.328598] Key type .llcrypt unregistered [ 1543.329598] Key type ._llcrypt unregistered [ 1550.185616] Key type ._llcrypt registered [ 1550.186596] Key type .llcrypt registered [ 1550.219773] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1550.555207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1550.612901] alg: No test for adler32 (adler32-zlib) [ 1551.479225] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1551.561595] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1553.136115] Key type lgssc registered [ 1553.467507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1555.943141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1555.948730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1557.101741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1558.334241] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1561.181844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1561.290346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1562.590099] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1564.315093] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1565.511938] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1567.741542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1567.866877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1569.686971] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1571.423831] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1581.160291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1586.658477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1586.660269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1586.660470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1586.662678] Lustre: Skipped 1 previous similar message [ 1590.260410] LustreError: 108190:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1590.296540] Lustre: server umount lustre-OST0000 complete [ 1591.777440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1591.777986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1591.780621] Lustre: Skipped 1 previous similar message [ 1591.781794] Lustre: Skipped 1 previous similar message [ 1596.897170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1596.898877] Lustre: Skipped 2 previous similar messages [ 1597.694955] LustreError: 108393:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1597.696743] LustreError: 108393:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1597.725956] Lustre: server umount lustre-MDT0000 complete [ 1598.952081] LustreError: 108394:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877644 with bad export cookie 9358700889611172348 [ 1598.955226] LustreError: 108394:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1598.955765] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1599.021812] LustreError: 108595:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1599.023494] LustreError: 108595:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1599.070749] Lustre: server umount lustre-MDT0001 complete [ 1602.488143] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1603.446073] Key type lgssc unregistered [ 1603.564363] LNet: 109125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1603.566235] LNetError: 109125:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.581314] LNet: Removed LNI 192.168.204.127@tcp [ 1603.842706] Key type .llcrypt unregistered [ 1603.843726] Key type ._llcrypt unregistered [ 1610.402865] Key type ._llcrypt registered [ 1610.404048] Key type .llcrypt registered [ 1610.447940] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1610.936760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1610.950187] alg: No test for adler32 (adler32-zlib) [ 1611.808154] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1611.888283] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1613.480145] Key type lgssc registered [ 1613.866686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1615.930126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1615.932403] LustreError: 110179:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1615.941471] LustreError: 110181:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1615.945486] LustreError: 110179:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 1615.948432] LustreError: 110179:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1615.952655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.953910] LustreError: 110179:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1617.142884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1619.656264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1619.760711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1621.109953] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1622.669080] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing lsmod [ 1623.630147] LustreError: 111354:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1623.630189] LustreError: 111355:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1623.634621] LustreError: 111354:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1623.636521] LustreError: 111354:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1623.639619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1623.639759] LustreError: 111354:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1623.764432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1624.634326] LustreError: 110248:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1624.636461] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1624.637374] LustreError: 110248:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1624.637384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1624.640402] Lustre: Skipped 1 previous similar message [ 1624.645310] LustreError: 110248:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1629.665019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1629.666772] Lustre: Skipped 4 previous similar messages [ 1630.690441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1630.694378] Lustre: Skipped 2 previous similar messages [ 1630.787419] LustreError: 111543:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1630.818682] Lustre: server umount lustre-MDT0000 complete [ 1632.029370] LustreError: 110197:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877677 with bad export cookie 3782011275440252854 [ 1632.035362] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1632.036081] LustreError: 110197:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1632.736243] LustreError: 110653:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000017875d45 x1849559778082304/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1632.749233] LustreError: 111748:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1632.752651] LustreError: 111748:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1632.802953] Lustre: server umount lustre-MDT0001 complete [ 1644.018514] LustreError: 111949:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1644.020212] LustreError: 111949:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1644.055403] Lustre: server umount lustre-OST0000 complete [ 1646.701359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1646.845069] LustreError: 112424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1646.852425] LustreError: 112424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1646.865953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1648.046274] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1650.838958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1650.967605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1652.366995] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1654.197214] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1655.408301] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1657.524855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1659.428649] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1661.063646] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1672.999929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1678.306853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.307454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1678.308136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1678.310030] Lustre: Skipped 2 previous similar messages [ 1679.855090] LustreError: 114691:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1679.857093] LustreError: 114691:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1679.890262] Lustre: server umount lustre-OST0000 complete [ 1681.888832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1681.891571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1683.346063] Lustre: server umount lustre-MDT0000 complete [ 1683.426203] LustreError: 112424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1683.432147] LustreError: 112424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1684.594146] LustreError: 112404:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877730 with bad export cookie 3782011275440253533 [ 1684.597465] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1684.598741] LustreError: 112404:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1684.674337] LustreError: 115095:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1684.676953] LustreError: 115095:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 1684.730554] Lustre: server umount lustre-MDT0001 complete [ 1688.542928] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1689.528957] Key type lgssc unregistered [ 1689.649398] LNet: 115626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1689.651619] LNetError: 115626:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1689.662271] LNet: Removed LNI 192.168.204.127@tcp [ 1689.955535] Key type .llcrypt unregistered [ 1689.957332] Key type ._llcrypt unregistered [ 1699.029588] Key type ._llcrypt registered [ 1699.030567] Key type .llcrypt registered [ 1699.063613] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1699.455693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1699.479872] alg: No test for adler32 (adler32-zlib) [ 1700.324904] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1700.402736] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1701.984119] Key type lgssc registered [ 1702.318183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1704.794111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1704.800270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.956255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1707.235335] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1710.062893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1710.188436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1711.536844] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1713.269620] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1714.457701] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1716.659688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1716.761016] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1718.519461] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1720.263971] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1731.109603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1737.823590] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1738.650472] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1741.282954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1741.284043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.287121] LustreError: Skipped 1 previous similar message [ 1741.291226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1741.294838] Lustre: Skipped 1 previous similar message [ 1745.711912] LustreError: 119367:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1745.747334] Lustre: server umount lustre-OST0000 complete [ 1751.008857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1751.010989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.014166] Lustre: Skipped 1 previous similar message [ 1751.015761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1753.156706] LustreError: 119573:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1753.159971] LustreError: 119573:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1753.194277] Lustre: server umount lustre-MDT0000 complete [ 1754.411282] LustreError: 119281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877800 with bad export cookie 6176620679554986284 [ 1754.414553] LustreError: 119281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1754.415365] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1754.480371] LustreError: 119775:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1754.482499] LustreError: 119775:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1754.531592] Lustre: server umount lustre-MDT0001 complete [ 1758.208457] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1759.256391] Key type lgssc unregistered [ 1759.387410] LNet: 120307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.389680] LNetError: 120307:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.400298] LNet: Removed LNI 192.168.204.127@tcp [ 1759.716957] Key type .llcrypt unregistered [ 1759.718785] Key type ._llcrypt unregistered [ 1766.370264] Key type ._llcrypt registered [ 1766.371967] Key type .llcrypt registered [ 1766.411167] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1766.766265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1766.834260] alg: No test for adler32 (adler32-zlib) [ 1767.684815] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1767.761817] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1769.336183] Key type lgssc registered [ 1769.704794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1772.292765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1772.300061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1773.457229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1774.692181] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1777.422772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1777.535658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1778.797253] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1780.503624] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1781.644198] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1783.883929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1783.994241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1785.811941] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1787.586954] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1798.314281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1803.746443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1803.746772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1803.747932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1803.749747] Lustre: Skipped 1 previous similar message [ 1807.022103] LustreError: 123753:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1807.053987] Lustre: server umount lustre-OST0000 complete [ 1808.352824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1808.356166] LustreError: Skipped 1 previous similar message [ 1808.358042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1808.362057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1808.363891] Lustre: Skipped 1 previous similar message [ 1813.985790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1813.987908] Lustre: Skipped 2 previous similar messages [ 1814.338082] LustreError: 123955:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1814.339860] LustreError: 123955:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1814.371600] Lustre: server umount lustre-MDT0000 complete [ 1815.621362] LustreError: 123956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877861 with bad export cookie 16866610703315664505 [ 1815.625986] LustreError: 123956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1815.627067] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1815.692815] LustreError: 124157:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1815.695374] LustreError: 124157:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1815.746530] Lustre: server umount lustre-MDT0001 complete [ 1819.233961] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1820.251628] Key type lgssc unregistered [ 1820.367406] LNet: 124688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.370218] LNetError: 124688:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.378313] LNet: Removed LNI 192.168.204.127@tcp [ 1820.683196] Key type .llcrypt unregistered [ 1820.684260] Key type ._llcrypt unregistered [ 1829.389624] Key type ._llcrypt registered [ 1829.390680] Key type .llcrypt registered [ 1829.433803] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1829.859312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1829.887790] alg: No test for adler32 (adler32-zlib) [ 1830.752583] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1830.839649] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1832.424250] Key type lgssc registered [ 1832.812224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1835.330785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1835.338481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1836.483772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1837.655029] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1840.382113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1840.511571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1841.780845] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1843.511877] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1844.600804] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1846.767291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1846.887806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.708559] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1850.410517] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1861.222480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 1868.354038] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1869.539055] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1869.541370] Lustre: Skipped 2 previous similar messages [ 1870.861544] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1870.863578] Lustre: Skipped 2 previous similar messages [ 1878.007037] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1878.016496] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1880.796239] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.27@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1880.799023] Lustre: Skipped 2 previous similar messages [ 1887.740132] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.27@tcp 192.168.0.[2,10]@tcp [ 1889.848612] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.27/24@tcp in log lustre-MDT0000 [ 1889.851229] Lustre: Skipped 2 previous similar messages [ 1897.973443] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.27/24@tcp [ 1897.975254] Lustre: Skipped 1 previous similar message [ 1902.562954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1902.565042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.568094] LustreError: Skipped 1 previous similar message [ 1902.569256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1902.570835] Lustre: Skipped 1 previous similar message [ 1906.350587] LustreError: 130265:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1906.386301] Lustre: server umount lustre-OST0000 complete [ 1907.681409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1907.684663] Lustre: Skipped 1 previous similar message [ 1907.686559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1912.289025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1912.291922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1912.295632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1913.732300] LustreError: 130466:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1913.735282] LustreError: 130466:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1913.768522] Lustre: server umount lustre-MDT0000 complete [ 1914.980907] LustreError: 126041:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877960 with bad export cookie 18216346523536148064 [ 1914.983634] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1914.983959] LustreError: 126041:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1915.061080] LustreError: 130667:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1915.062538] LustreError: 130667:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1915.119974] Lustre: server umount lustre-MDT0001 complete [ 1918.696717] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1919.709408] Key type lgssc unregistered [ 1919.827334] LNet: 131198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.829352] LNetError: 131198:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.839309] LNet: Removed LNI 192.168.204.127@tcp [ 1920.118490] Key type .llcrypt unregistered [ 1920.119426] Key type ._llcrypt unregistered [ 1930.229629] Key type ._llcrypt registered [ 1930.231309] Key type .llcrypt registered [ 1930.272890] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1930.789812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1930.800778] alg: No test for adler32 (adler32-zlib) [ 1931.665820] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1931.760979] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1933.352220] Key type lgssc registered [ 1933.743461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1936.567429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1936.577363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1937.739245] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1937.741156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1937.749553] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.27/24@tcp [ 1939.175855] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1942.967219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1943.087889] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1943.098693] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.27/24@tcp [ 1943.116234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1944.570385] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1946.423288] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1947.564688] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1949.847612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1950.024867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1951.988202] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1954.090896] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1962.347250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 1968.609343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1968.613077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1968.620478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1972.709926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1972.711205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1972.711604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.714047] Lustre: Skipped 1 previous similar message [ 1973.806085] LustreError: 135043:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1973.847736] Lustre: server umount lustre-OST0000 complete [ 1977.826029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.826582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1977.830988] Lustre: Skipped 1 previous similar message [ 1977.833379] Lustre: Skipped 1 previous similar message [ 1981.314967] LustreError: 135244:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1981.316603] LustreError: 135244:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1981.344519] Lustre: server umount lustre-MDT0000 complete [ 1982.562684] LustreError: 132698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878028 with bad export cookie 1471425769386855153 [ 1982.568523] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1982.570156] LustreError: 132698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1982.646058] LustreError: 135447:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1982.647837] LustreError: 135447:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1982.708808] Lustre: server umount lustre-MDT0001 complete [ 1986.345910] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1987.414202] Key type lgssc unregistered [ 1987.546497] LNet: 135978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1987.550778] LNetError: 135978:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1987.560406] LNet: Removed LNI 192.168.204.127@tcp [ 1987.872518] Key type .llcrypt unregistered [ 1987.873603] Key type ._llcrypt unregistered [ 1996.223392] Key type ._llcrypt registered [ 1996.224385] Key type .llcrypt registered [ 1996.258612] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 2002.155433] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2002.523694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2002.539697] alg: No test for adler32 (adler32-zlib) [ 2003.385568] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 2003.462809] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 2005.040211] Key type lgssc registered [ 2005.372213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2008.128797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2010.211461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2012.316813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2014.291888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2016.854507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2016.877328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2016.883317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.970994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2017.982433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2018.019317] Lustre: lustre-MDT0000: new disk, initializing [ 2018.045024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2018.051608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2019.537078] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2024.271969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2024.296577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2024.333175] Lustre: 139327:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2024.354134] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2024.355768] Lustre: Skipped 1 previous similar message [ 2024.389035] Lustre: lustre-MDT0001: new disk, initializing [ 2024.417810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2024.428558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2024.431407] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2025.621526] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2028.641648] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2029.718347] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2031.865612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2031.891268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2031.983904] Lustre: lustre-OST0000: new disk, initializing [ 2031.986810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2032.016396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2032.304267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2032.309715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2032.344457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2033.785031] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2036.817648] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2038.121539] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2038.182519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2039.385383] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2039.449485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2042.850715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2042.851162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.853892] LustreError: Skipped 1 previous similar message [ 2042.858072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2042.862498] Lustre: Skipped 1 previous similar message [ 2046.127118] LustreError: 142266:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2046.165862] Lustre: server umount lustre-OST0000 complete [ 2047.969375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.969954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2047.972495] Lustre: Skipped 2 previous similar messages [ 2053.089354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2053.092269] Lustre: Skipped 2 previous similar messages [ 2053.568064] LustreError: 142467:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2053.569987] LustreError: 142467:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2053.601433] Lustre: server umount lustre-MDT0000 complete [ 2054.773182] LustreError: 139319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878100 with bad export cookie 16320529202582168066 [ 2054.776552] LustreError: 139319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2054.777278] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2054.840902] LustreError: 142669:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2054.843520] LustreError: 142669:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2054.896787] Lustre: server umount lustre-MDT0001 complete [ 2057.582895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2057.753581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2058.916113] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2062.477042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2062.596558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2064.076649] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2065.964217] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2067.163388] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2069.955645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2072.226436] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2074.125662] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2075.729903] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2082.930671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 2084.197219] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2084.272609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2087.551241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2087.574159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2087.635248] Lustre: lustre-OST0001: new disk, initializing [ 2087.636957] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2087.671902] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2087.675626] Lustre: Skipped 1 previous similar message [ 2089.449450] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2092.644561] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2097.153909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2097.156561] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2097.183719] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2097.804790] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 2099.014354] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2099.079531] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2100.199599] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d9b10133800.ost_server_uuid 50 [ 2100.716558] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d9b10133800.ost_server_uuid in FULL state after 0 sec [ 2107.361702] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2107.363776] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2107.367740] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2107.370698] Lustre: Skipped 1 previous similar message [ 2108.590082] LustreError: 147475:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2108.591813] LustreError: 147475:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2108.608625] Lustre: server umount lustre-OST0001 complete [ 2112.484974] LustreError: 144740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2112.485826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.488615] LustreError: 144740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2112.493833] Lustre: Skipped 1 previous similar message [ 2112.495820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2112.497257] Lustre: Skipped 1 previous similar message [ 2117.605672] LustreError: 147389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.610021] LustreError: 147389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2122.722889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2122.723487] LustreError: 145066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.724803] Lustre: Skipped 3 previous similar messages [ 2122.728756] LustreError: 145066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2124.256097] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2124.277081] LustreError: 147676:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2124.278909] LustreError: 147676:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2124.317787] Lustre: server umount lustre-OST0000 complete [ 2127.841977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.845342] Lustre: Skipped 1 previous similar message [ 2131.680935] Lustre: server umount lustre-MDT0000 complete [ 2132.908971] LustreError: 143169:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878178 with bad export cookie 16320529202582168934 [ 2132.912379] LustreError: 143169:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2132.913493] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2132.985211] LustreError: 148080:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2132.986982] LustreError: 148080:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2133.054844] Lustre: server umount lustre-MDT0001 complete [ 2139.169261] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 2140.229585] Key type lgssc unregistered [ 2140.363400] LNet: 149067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2140.365445] LNetError: 149067:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2140.371491] LNet: Removed LNI 192.168.204.127@tcp [ 2140.678107] Key type .llcrypt unregistered [ 2140.679105] Key type ._llcrypt unregistered [ 2151.170414] Key type ._llcrypt registered [ 2151.171533] Key type .llcrypt registered [ 2151.207702] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 2157.024613] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2157.328604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2157.370847] alg: No test for adler32 (adler32-zlib) [ 2158.221591] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 2158.297372] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 2159.872210] Key type lgssc registered [ 2160.200416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2162.913992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2164.981637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2166.854870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2168.831150] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2170.871823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2170.889927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2170.894498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2171.972309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2171.985392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2172.019344] Lustre: lustre-MDT0000: new disk, initializing [ 2172.041800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.047308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2173.313626] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2177.346485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2177.368988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2177.403281] Lustre: 152988:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2177.430254] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2177.432063] Lustre: Skipped 1 previous similar message [ 2177.467492] Lustre: lustre-MDT0001: new disk, initializing [ 2177.501377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2177.515175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2177.519700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2178.706792] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2181.690344] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2182.929915] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2185.037871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2185.060634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2185.160505] Lustre: lustre-OST0000: new disk, initializing [ 2185.162339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2185.194343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2187.032338] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2187.327515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2187.330870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2187.376079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2189.999588] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2191.215308] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2191.288826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2192.430503] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2192.500259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2197.475093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2197.475699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2197.478883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2197.482803] Lustre: Skipped 1 previous similar message [ 2199.149603] LustreError: 155923:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2199.181765] Lustre: server umount lustre-OST0000 complete [ 2202.594898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2202.595671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2202.600936] Lustre: Skipped 2 previous similar messages [ 2207.713868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2207.716886] Lustre: Skipped 2 previous similar messages [ 2212.833510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2212.835245] Lustre: Skipped 1 previous similar message [ 2215.392074] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2215.427766] LustreError: 156125:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2215.429632] LustreError: 156125:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2215.464105] Lustre: server umount lustre-MDT0000 complete [ 2216.603346] LustreError: 152980:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878262 with bad export cookie 15452092583572055086 [ 2216.606473] LustreError: 152980:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2216.606757] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2216.672505] LustreError: 156329:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2216.674210] LustreError: 156329:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2216.721789] Lustre: server umount lustre-MDT0001 complete [ 2219.396117] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 01:11:05 (1763878265) [ 2224.079662] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 2226.915237] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2229.791628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2231.751516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2233.483831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2235.303053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2237.380683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2237.406019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2237.486125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2237.496301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2237.531947] Lustre: lustre-MDT0000: new disk, initializing [ 2237.553418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2237.559221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2238.830810] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2242.785878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2242.811344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2242.848991] Lustre: 159314:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2242.873339] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2242.875077] Lustre: Skipped 1 previous similar message [ 2242.909463] Lustre: lustre-MDT0001: new disk, initializing [ 2242.940772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2242.952124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2242.955221] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2244.069730] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2246.908341] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2247.952953] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2250.073863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2250.095356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2251.911173] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2254.813887] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2256.372696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2256.375491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2256.400994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2261.916107] Lustre: Failing over lustre-OST0000 [ 2261.948060] LustreError: 161754:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2261.949788] LustreError: 161754:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2261.985126] Lustre: server umount lustre-OST0000 complete [ 2274.615223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2274.727531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2274.731920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2274.732352] Lustre: Skipped 1 previous similar message [ 2274.734189] Lustre: Skipped 1 previous similar message [ 2274.737445] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2274.739216] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2276.546345] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2277.815643] Lustre: Failing over lustre-MDT0000 [ 2277.822303] LustreError: 151203:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2277.888182] LustreError: 162726:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2277.891543] LustreError: 162726:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2277.924600] Lustre: server umount lustre-MDT0000 complete [ 2278.880853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2278.884405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.914765] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 2279.914801] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2285.025521] LustreError: 160100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2285.029082] LustreError: 160100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2286.813444] LustreError: 159318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.096111] Lustre: 151206:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763878317/real 1763878317] req@000000000d0da946 x1849560351697280/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763878333 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2290.146354] LustreError: 160100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2290.153460] LustreError: 160100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2290.818547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2290.856993] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2290.861028] Lustre: 159316:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.204.127@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2290.938283] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 2290.964033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2290.986762] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2291.934048] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2292.219766] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2296.293786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2296.305577] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2296.321563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2300.384888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2300.388507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.393011] Lustre: Skipped 2 previous similar messages [ 2300.395492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2303.407070] LustreError: 163611:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2303.408873] LustreError: 163611:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2303.442227] Lustre: server umount lustre-OST0000 complete [ 2306.529905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2306.531821] LustreError: Skipped 1 previous similar message [ 2310.744514] Lustre: server umount lustre-MDT0000 complete [ 2311.649168] LustreError: 159323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2311.653858] LustreError: 159323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2311.958276] LustreError: 159306:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878357 with bad export cookie 15452092583572057144 [ 2311.961197] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2311.961532] LustreError: 159306:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2312.079615] Lustre: server umount lustre-MDT0001 complete [ 2315.427439] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 2316.266818] Key type lgssc unregistered [ 2316.385349] LNet: 164546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2316.387365] LNetError: 164546:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2316.395288] LNet: Removed LNI 192.168.204.127@tcp [ 2316.633624] Key type .llcrypt unregistered [ 2316.634577] Key type ._llcrypt unregistered [ 2335.732966] Key type ._llcrypt registered [ 2335.733948] Key type .llcrypt registered [ 2335.765127] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2336.064269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2336.120620] alg: No test for adler32 (adler32-zlib) [ 2336.959423] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 2337.031311] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 2338.608107] Key type lgssc registered [ 2338.959527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2341.325020] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2341.330285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2342.474649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2343.577830] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2346.531531] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2348.444130] LustreError: 166072:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2348.446968] LustreError: 166072:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2348.450235] LustreError: 166072:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2348.508882] LustreError: 166595:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2348.556712] Lustre: server umount lustre-MDT0000 complete [ 2352.871382] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 2355.856728] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2358.911565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2360.999200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2362.991090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2364.969135] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2367.218697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2367.241281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.328996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2367.339269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2367.374174] Lustre: lustre-MDT0000: new disk, initializing [ 2367.397839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2367.404233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2368.604801] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2372.608917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2372.635367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2372.665115] Lustre: 169234:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2372.686694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2372.688286] Lustre: Skipped 1 previous similar message [ 2372.722550] Lustre: lustre-MDT0001: new disk, initializing [ 2372.751203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2372.762743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2372.765455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2373.911313] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2376.770386] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2377.847323] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2380.053247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2380.072499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2380.163005] Lustre: lustre-OST0000: new disk, initializing [ 2380.165500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2380.199961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2381.176127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2381.179188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2381.207419] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2381.933642] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2384.887299] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2386.665193] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2387.264819] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 01:13:53 (1763878433) [ 2447.840555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2447.840763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2447.843813] Lustre: Skipped 2 previous similar messages [ 2447.845334] Lustre: Skipped 1 previous similar message [ 2452.960829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2452.962985] Lustre: Skipped 2 previous similar messages [ 2458.080483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2458.082137] Lustre: Skipped 2 previous similar messages [ 2461.152136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2461.187416] LustreError: 172733:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2461.189056] LustreError: 172733:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2461.230778] Lustre: server umount lustre-MDT0000 complete [ 2462.416483] LustreError: 169224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878508 with bad export cookie 17568606350573412949 [ 2462.417680] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2462.420150] LustreError: 169224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2463.200446] LustreError: 169241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.200588] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2463.204366] LustreError: 169241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2463.207764] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2467.232555] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2467.232555] LustreError: 169241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.538957] LustreError: 172935:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2468.540765] LustreError: 172935:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2468.577179] Lustre: server umount lustre-MDT0001 complete [ 2484.192069] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2484.210630] LustreError: 173136:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2484.212305] LustreError: 173136:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2484.242562] Lustre: server umount lustre-OST0000 complete [ 2487.073542] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 2489.879489] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2492.787196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2494.697739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2496.508686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2498.446719] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2500.603557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2500.621503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2500.694610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2500.704192] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2500.738380] Lustre: lustre-MDT0000: new disk, initializing [ 2500.760380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2500.766052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2501.868884] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2505.593347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2505.614373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2505.642112] Lustre: 175470:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2505.661216] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2505.662751] Lustre: Skipped 1 previous similar message [ 2505.696467] Lustre: lustre-MDT0001: new disk, initializing [ 2505.733696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2505.737682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2506.816117] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2509.597902] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2510.642287] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2512.664730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2512.684627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2512.794387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2512.796815] Lustre: Skipped 1 previous similar message [ 2514.036322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2514.039233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2514.069121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2514.503097] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2517.368186] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2518.531196] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2518.601225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2519.742578] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2519.805084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2522.415135] LustreError: 178407:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2522.416869] LustreError: 178407:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2522.448094] Lustre: server umount lustre-OST0000 complete [ 2524.641783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.642382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2524.645084] Lustre: Skipped 1 previous similar message [ 2533.859909] LustreError: 178609:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2533.861625] LustreError: 178609:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2533.888958] Lustre: server umount lustre-MDT0000 complete [ 2534.881437] LustreError: 175482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.885556] LustreError: 175482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2535.069497] LustreError: 175461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878580 with bad export cookie 17568606350573571765 [ 2535.074095] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2544.663198] Lustre: server umount lustre-MDT0001 complete [ 2547.301728] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:16:33 (1763878593) [ 2551.878601] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 2554.756278] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2557.663718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2559.666581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2561.526554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2563.434929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2565.560129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2565.586713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.670362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2565.681837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2565.683506] Lustre: Skipped 1 previous similar message [ 2565.717296] Lustre: lustre-MDT0000: new disk, initializing [ 2565.718555] Lustre: Skipped 1 previous similar message [ 2565.739721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.745383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2566.909106] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2570.718639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2570.739827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2570.770113] Lustre: 181796:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2570.859837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2571.980156] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2574.802069] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2575.907242] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2578.072544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2578.095439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2579.944907] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2581.816966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2581.854333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2582.841419] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2587.106881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2587.107103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2587.108932] LustreError: Skipped 1 previous similar message [ 2587.112419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2587.114375] Lustre: Skipped 4 previous similar messages [ 2590.575541] LustreError: 184337:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2590.577309] LustreError: 184337:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2590.608616] Lustre: server umount lustre-OST0000 complete [ 2592.225619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.228743] Lustre: Skipped 2 previous similar messages [ 2606.048084] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2606.109560] Lustre: server umount lustre-MDT0000 complete [ 2607.273196] LustreError: 182579:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878653 with bad export cookie 17568606350573572633 [ 2607.275969] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2607.276858] LustreError: 182579:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2607.383580] Lustre: server umount lustre-MDT0001 complete [ 2609.940261] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:17:35 (1763878655) [ 2615.621879] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 2618.504112] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2621.286644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2623.191711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2624.999182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2626.947967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2629.055169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2629.079372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.158036] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2629.168944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2629.170566] Lustre: Skipped 3 previous similar messages [ 2629.207301] Lustre: lustre-MDT0000: new disk, initializing [ 2629.209593] Lustre: Skipped 2 previous similar messages [ 2629.232249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2629.233852] Lustre: Skipped 2 previous similar messages [ 2629.238671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2629.240650] Lustre: Skipped 2 previous similar messages [ 2630.352985] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2634.196539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2634.220437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2634.254281] Lustre: 187728:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2634.356063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2635.520913] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2638.360693] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2639.421129] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2641.454797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2641.482567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2643.278379] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2646.051766] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2646.240257] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2651.108470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2651.109654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2651.113455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2651.114377] LustreError: Skipped 1 previous similar message [ 2651.117489] Lustre: Skipped 7 previous similar messages [ 2654.128138] LustreError: 190270:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2654.130225] LustreError: 190270:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2654.170905] Lustre: server umount lustre-OST0000 complete [ 2661.463346] Lustre: server umount lustre-MDT0000 complete [ 2662.609028] LustreError: 187720:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878708 with bad export cookie 17568606350573573704 [ 2662.612111] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2662.614587] LustreError: 187720:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2662.719967] Lustre: server umount lustre-MDT0001 complete [ 2666.423611] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 2667.332691] Key type lgssc unregistered [ 2667.453429] LNet: 191204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2667.457080] LNetError: 191204:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2667.467583] LNet: Removed LNI 192.168.204.127@tcp [ 2667.819430] Key type .llcrypt unregistered [ 2667.820701] Key type ._llcrypt unregistered [ 2676.005882] Key type ._llcrypt registered [ 2676.006855] Key type .llcrypt registered [ 2676.039371] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2676.421535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2676.433854] alg: No test for adler32 (adler32-zlib) [ 2677.270925] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 2677.346204] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 2678.920136] Key type lgssc registered [ 2679.237358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2681.541716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2681.547344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2682.697950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2683.858753] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2686.412314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2686.535296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2687.825439] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2689.465720] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2690.655563] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2692.830465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2692.957910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2694.681589] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2696.351498] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2707.303994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2727.906131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2727.906288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2727.908237] LustreError: Skipped 1 previous similar message [ 2727.911672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2727.913799] Lustre: Skipped 1 previous similar message [ 2729.263134] LustreError: 194849:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2729.302504] Lustre: server umount lustre-OST0000 complete [ 2732.512996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2732.515987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.520389] Lustre: Skipped 1 previous similar message [ 2732.522368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2736.575154] LustreError: 195050:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2736.576993] LustreError: 195050:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2736.601124] Lustre: server umount lustre-MDT0000 complete [ 2737.751401] LustreError: 192558:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878783 with bad export cookie 15721730974849471933 [ 2737.753958] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2737.754289] LustreError: 192558:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2737.824540] LustreError: 195253:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2737.826135] LustreError: 195253:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2737.874155] Lustre: server umount lustre-MDT0001 complete [ 2740.811854] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 2741.733187] Key type lgssc unregistered [ 2741.850390] LNet: 195783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2741.852394] LNetError: 195783:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2741.862319] LNet: Removed LNI 192.168.204.127@tcp [ 2742.122617] Key type .llcrypt unregistered [ 2742.123553] Key type ._llcrypt unregistered [ 2750.027445] Key type ._llcrypt registered [ 2750.028390] Key type .llcrypt registered [ 2750.065134] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2750.458237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2750.528676] alg: No test for adler32 (adler32-zlib) [ 2751.375172] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 2751.461995] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 2753.040096] Key type lgssc registered [ 2753.401787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2755.949414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2755.956588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2757.157426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2758.384655] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2761.140901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2761.265575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2762.543312] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2764.399933] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2765.578930] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2767.757682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2767.887784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2769.864321] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2771.888983] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2782.243965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2787.301742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2787.302264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2787.303890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2787.305306] Lustre: Skipped 1 previous similar message [ 2790.639388] LustreError: 199420:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2790.680082] Lustre: server umount lustre-OST0000 complete [ 2792.422719] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9d9b02e02800.ost_server_uuid 50 [ 2792.946439] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9d9b02e02800.ost_server_uuid in DISCONN state after 0 sec [ 2793.410924] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2807.265093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2807.267060] LustreError: Skipped 1 previous similar message [ 2807.268086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2807.271427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2807.273135] Lustre: Skipped 2 previous similar messages [ 2808.801942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2808.804761] Lustre: Skipped 2 previous similar messages [ 2809.472648] LustreError: 199817:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2809.474618] LustreError: 199817:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2809.500706] Lustre: server umount lustre-MDT0000 complete [ 2810.691897] LustreError: 197138:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878856 with bad export cookie 9914282579579992729 [ 2810.695974] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2810.698479] LustreError: 197138:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2810.776658] LustreError: 200019:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2810.779036] LustreError: 200019:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2810.823867] Lustre: server umount lustre-MDT0001 complete [ 2813.367810] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 01:20:59 (1763878859) [ 2815.366630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.505808] LustreError: 200688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2815.510409] LustreError: 200688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2815.526353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2816.658070] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2819.195790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2820.461760] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2822.138521] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2823.232274] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2825.305874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2825.422327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2825.424035] Lustre: Skipped 1 previous similar message [ 2826.474974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2827.136790] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2828.748535] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2830.838240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2830.866704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2830.930880] Lustre: lustre-OST0001: new disk, initializing [ 2830.932655] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2832.666480] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2833.085391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2833.090151] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2833.127042] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2835.572282] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2836.960715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2836.962690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2836.965365] Lustre: Skipped 1 previous similar message [ 2836.966775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2842.479134] LustreError: 203893:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2842.480812] LustreError: 203893:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2842.497380] Lustre: server umount lustre-OST0000 complete [ 2843.624577] LustreError: 202287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2843.631340] LustreError: 202287:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2844.288530] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2844.360217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2845.595166] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2845.667736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2845.918842] LustreError: 202287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2848.744081] LustreError: 202286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2848.748530] LustreError: 202286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2853.862241] LustreError: 202286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.869871] LustreError: 202286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2858.979891] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2858.980763] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2858.982966] Lustre: Skipped 1 previous similar message [ 2858.983572] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2858.988165] Lustre: Skipped 4 previous similar messages [ 2861.678152] LustreError: 204596:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2861.680154] LustreError: 204596:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2861.716972] Lustre: server umount lustre-OST0001 complete [ 2864.099455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2864.102501] Lustre: Skipped 2 previous similar messages [ 2869.078483] Lustre: server umount lustre-MDT0000 complete [ 2869.217135] LustreError: 200688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.220770] LustreError: 200688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2870.240286] LustreError: 200713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878916 with bad export cookie 9914282579579993786 [ 2870.242962] LustreError: 200713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2870.243641] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2870.308716] LustreError: 204998:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2870.310770] LustreError: 204998:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2870.359426] Lustre: server umount lustre-MDT0001 complete [ 2872.097516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2873.327832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2874.526711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2875.755889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2877.677854] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 01:22:03 (1763878923) [ 2879.732299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2879.841161] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2879.855990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2879.927106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2879.928801] Lustre: Skipped 1 previous similar message [ 2881.162905] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2883.852618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2883.881899] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2883.894911] Lustre: 206486:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2885.205941] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2886.889367] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2888.011986] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2890.156373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2890.233900] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2892.026316] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2893.682507] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2894.878418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 2895.892667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2895.914977] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2895.997547] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2895.999351] Lustre: Skipped 2 previous similar messages [ 2897.785640] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2899.479258] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2901.476902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.478184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2901.479877] Lustre: Skipped 1 previous similar message [ 2901.481395] Lustre: Skipped 2 previous similar messages [ 2906.348118] LustreError: 209670:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2906.351329] LustreError: 209670:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2906.367425] Lustre: server umount lustre-OST0000 complete [ 2906.595857] LustreError: 208114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2906.600510] LustreError: 208114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2921.956537] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2921.956696] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.958612] LustreError: Skipped 2 previous similar messages [ 2921.962156] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2921.964596] Lustre: Skipped 2 previous similar messages [ 2922.953665] Lustre: server umount lustre-OST0001 complete [ 2938.336203] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2938.379843] LustreError: 210085:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2938.381752] LustreError: 210085:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2938.406125] Lustre: server umount lustre-MDT0000 complete [ 2939.552079] LustreError: 206477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878985 with bad export cookie 9914282579579995144 [ 2939.554691] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2939.555081] LustreError: 206477:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2939.667312] Lustre: server umount lustre-MDT0001 complete [ 2941.358308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2942.534500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2943.665965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2944.805448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2946.859464] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 01:23:12 (1763878992) [ 2951.747788] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 2954.791286] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2958.032100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2959.977656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2961.781790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2963.722358] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2965.840738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2965.859986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.936177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2965.945773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2965.980462] Lustre: lustre-MDT0000: new disk, initializing [ 2966.001911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2966.007603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2967.145567] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2971.031624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2971.054206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2971.086518] Lustre: 214104:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2971.109081] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2971.110655] Lustre: Skipped 1 previous similar message [ 2971.143895] Lustre: lustre-MDT0001: new disk, initializing [ 2971.184157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2971.186926] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2972.338916] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2975.249869] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2976.296127] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2978.486638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2978.518294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2978.621654] Lustre: lustre-OST0000: new disk, initializing [ 2978.623437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2980.416732] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2981.436265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2981.439332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2981.480116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2983.375376] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2984.673790] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2984.738519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2985.919219] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2985.980278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2986.977137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2986.979030] LustreError: Skipped 1 previous similar message [ 2986.980151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2986.982928] Lustre: Skipped 3 previous similar messages [ 2986.984513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2986.985857] Lustre: Skipped 7 previous similar messages [ 2988.683765] Lustre: server umount lustre-OST0000 complete [ 3004.384122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3004.424535] LustreError: 217244:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3004.427322] LustreError: 217244:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3004.456596] Lustre: server umount lustre-MDT0000 complete [ 3005.671344] LustreError: 214884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879051 with bad export cookie 9914282579579996663 [ 3005.676150] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3016.213951] Lustre: server umount lustre-MDT0001 complete [ 3018.917417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3019.071149] LustreError: 217923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3019.075911] LustreError: 217923:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3020.307926] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3023.067901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3024.433932] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3026.073097] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3027.224337] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3029.382122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3031.185283] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3032.819460] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3034.028681] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3044.319960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3045.566577] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3045.635700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3048.932070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3048.934176] LustreError: Skipped 1 previous similar message [ 3052.371933] Lustre: server umount lustre-OST0000 complete [ 3054.173502] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3055.366753] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3078.690126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3078.790381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3078.794316] Lustre: Skipped 5 previous similar messages [ 3080.454992] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3082.068536] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3082.601212] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3088.612036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3088.613820] Lustre: Skipped 9 previous similar messages [ 3089.607752] Lustre: server umount lustre-OST0000 complete [ 3093.986408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3093.989418] Lustre: Skipped 5 previous similar messages [ 3105.248128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3105.335593] Lustre: server umount lustre-MDT0000 complete [ 3106.697249] LustreError: 217903:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879152 with bad export cookie 9914282579579997531 [ 3106.701135] LustreError: 217903:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3106.703435] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3106.838949] Lustre: server umount lustre-MDT0001 complete [ 3110.162233] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 01:25:56 (1763879156) [ 3112.730958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3112.899306] LustreError: 223445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3114.339371] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3117.369127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3118.711020] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3120.582462] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3121.801827] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3123.993284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3125.912869] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3127.619664] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3128.896978] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3138.110897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3139.239165] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3139.299417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3141.435429] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3141.474271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3141.547693] Lustre: lustre-OST0001: new disk, initializing [ 3141.549598] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3143.605993] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3146.580487] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3147.866567] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3150.357091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3150.364344] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3150.409672] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3151.005705] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3152.321200] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3152.404886] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3155.432595] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3155.435928] LustreError: Skipped 1 previous similar message [ 3159.085135] LustreError: 227724:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3159.087094] LustreError: 227724:0:(obd_class.h:479:obd_check_dev()) Skipped 35 previous similar messages [ 3159.105231] Lustre: server umount lustre-OST0001 complete [ 3161.118892] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3161.196491] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3162.770487] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3162.863733] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3186.569583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3188.675681] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3190.865391] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3191.703291] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3191.785479] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3191.789856] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3191.790214] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3198.522588] Lustre: server umount lustre-OST0001 complete [ 3216.593639] Lustre: server umount lustre-OST0000 complete [ 3219.937240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3219.939692] LustreError: Skipped 2 previous similar messages [ 3219.941898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3219.943763] Lustre: Skipped 13 previous similar messages [ 3222.499695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3222.507823] Lustre: Skipped 7 previous similar messages [ 3224.221236] Lustre: server umount lustre-MDT0000 complete [ 3225.664398] LustreError: 223424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879271 with bad export cookie 9914282579579998462 [ 3225.667982] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3225.670322] LustreError: 223424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3231.868956] Lustre: server umount lustre-MDT0001 complete [ 3233.989625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3235.581741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3237.345923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3238.972657] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3241.995174] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:28:07 (1763879287) [ 3244.663859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3244.795301] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3244.805679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3244.886131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3244.889441] Lustre: Skipped 5 previous similar messages [ 3246.348708] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3249.621703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3249.660129] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3249.670543] Lustre: 231504:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3251.011385] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3253.058863] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3254.622675] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3256.956942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3257.048542] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3259.416525] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3261.466220] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3265.541922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3265.571807] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3267.680111] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3269.873454] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3270.454155] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3271.943311] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3272.034693] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3273.575373] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3273.686884] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3274.882118] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d9b07a97000.ost_server_uuid 50 [ 3275.348137] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d9b07a97000.ost_server_uuid in FULL state after 0 sec [ 3275.680099] Lustre: Permanently deactivating lustre-OST0001 [ 3275.685731] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3276.554301] Lustre: Permanently reactivating lustre-OST0001 [ 3283.711939] Lustre: server umount lustre-OST0001 complete [ 3286.505196] LustreError: 233130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3286.514319] LustreError: 233130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 3291.303630] Lustre: server umount lustre-OST0000 complete [ 3299.045718] Lustre: server umount lustre-MDT0000 complete [ 3300.412881] LustreError: 231541:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879346 with bad export cookie 9914282579579999736 [ 3300.419128] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3300.419849] LustreError: 231541:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3300.579375] Lustre: server umount lustre-MDT0001 complete [ 3302.662241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3303.938259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3305.140687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3306.376751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3308.229872] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 01:29:14 (1763879354) [ 3308.583415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3310.932709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.030448] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3311.040914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3311.043966] Lustre: Skipped 5 previous similar messages [ 3312.482328] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3315.594054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3315.632476] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3315.642262] Lustre: 237642:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3317.485946] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3319.755127] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3321.214470] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3324.081498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3324.122703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3324.238432] Lustre: Permanently deactivating lustre-OST0000 [ 3324.247610] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3326.579805] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3328.594875] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3329.537941] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3329.543824] LustreError: 239909:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 3329.548583] LustreError: 239909:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 3331.473666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3334.366738] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3336.670601] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3338.312672] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3342.413468] Lustre: Permanently reactivating lustre-OST0000 [ 3342.419028] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 3342.421087] Lustre: Skipped 2 previous similar messages [ 3345.270187] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3345.272952] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3345.276585] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3345.278601] Lustre: Skipped 1 previous similar message [ 3345.288455] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3345.294890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3352.035490] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 3352.037922] LustreError: Skipped 3 previous similar messages [ 3355.455193] Lustre: server umount lustre-OST0001 complete [ 3362.710563] Lustre: server umount lustre-OST0000 complete [ 3370.355664] Lustre: server umount lustre-MDT0000 complete [ 3371.915375] LustreError: 237633:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879417 with bad export cookie 9914282579580001374 [ 3371.923310] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3371.925237] LustreError: 237633:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3372.085383] Lustre: server umount lustre-MDT0001 complete [ 3376.083268] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 3377.182890] Key type lgssc unregistered [ 3377.324495] LNet: 242034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3377.327323] LNetError: 242034:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3377.337495] LNet: Removed LNI 192.168.204.127@tcp [ 3377.721155] Key type .llcrypt unregistered [ 3377.723198] Key type ._llcrypt unregistered [ 3387.535345] Key type ._llcrypt registered [ 3387.536408] Key type .llcrypt registered [ 3387.571577] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 3387.874634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3387.896071] alg: No test for adler32 (adler32-zlib) [ 3388.756860] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 3388.861352] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 3390.456199] Key type lgssc registered [ 3390.876728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3392.149571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3394.953940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3394.966786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3396.191192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3397.825326] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3400.652766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3400.680818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3400.718259] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3400.732351] Lustre: 243447:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3400.738051] Lustre: Permanently deactivating lustre-MDT0001 [ 3400.746060] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3400.870092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3402.188497] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3404.118249] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3405.477558] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3406.324039] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3407.740305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3407.863095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3409.514537] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3410.917363] LustreError: 245079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3410.920260] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3410.923531] LustreError: 245079:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3411.071201] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3411.937979] LustreError: 245590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3413.331184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3413.396770] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3415.479211] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3416.246708] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3417.576378] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3422.376239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3422.376296] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3423.323457] Lustre: Permanently reactivating lustre-MDT0001 [ 3423.328587] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3423.330867] Lustre: Skipped 2 previous similar messages [ 3423.356129] Lustre: lustre-MDT0001: Client d0955bcf-8846-4f0f-b9ec-d94d9c06c4d6 (at 192.168.204.27@tcp) reconnecting [ 3424.510412] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3424.577976] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3424.584115] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3424.587538] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3424.591871] LustreError: 246679:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3424.592639] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3424.595342] LustreError: 246679:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3424.600117] LustreError: 246679:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.127@tcp: cfg command failed: rc = -108 [ 3424.602879] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3424.602879] [ 3424.605448] LustreError: 243450:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3425.154224] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3426.867199] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3426.939645] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3428.188892] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3428.266462] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3429.170282] Lustre: Permanently deactivating lustre-MDT0001 [ 3429.178891] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3429.183676] Lustre: Skipped 2 previous similar messages [ 3431.807993] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3440.100709] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.102909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3440.104855] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3440.105280] Lustre: Skipped 3 previous similar messages [ 3440.118394] Lustre: Skipped 3 previous similar messages [ 3444.224949] LustreError: 247570:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3444.267149] Lustre: server umount lustre-MDT0000 complete [ 3445.218255] LustreError: 243458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3445.222802] LustreError: 243458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3445.518434] LustreError: 244182:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879491 with bad export cookie 15885426855649912811 [ 3445.521570] LustreError: 244182:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3445.524180] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3445.591676] LustreError: 247773:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3445.593526] LustreError: 247773:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3445.647543] Lustre: server umount lustre-MDT0001 complete [ 3452.911068] LustreError: 247974:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3452.913312] LustreError: 247974:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3452.928242] Lustre: server umount lustre-OST0000 complete [ 3456.693340] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 01:31:42 (1763879502) [ 3460.644266] LustreError: 248977:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3460.646736] LustreError: 248977:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3460.685081] Lustre: server umount lustre-OST0001 complete [ 3462.519723] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 3465.898454] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 3469.469251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3471.684689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3473.488562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3475.461498] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3477.978509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3478.007943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3478.095253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3478.099558] Lustre: Skipped 2 previous similar messages [ 3478.111913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3478.161569] Lustre: lustre-MDT0000: new disk, initializing [ 3478.187632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.194057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3479.674802] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3484.525530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3484.550048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3484.582582] Lustre: 251160:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3484.608758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3484.610854] Lustre: Skipped 1 previous similar message [ 3484.648394] Lustre: lustre-MDT0001: new disk, initializing [ 3484.702285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3484.707698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3486.141680] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3489.546291] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3491.027168] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3493.637636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3493.670363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3493.767091] Lustre: lustre-OST0000: new disk, initializing [ 3493.768993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3493.800542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3493.802636] Lustre: Skipped 1 previous similar message [ 3494.292272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3494.297405] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3494.336794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3496.117137] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3499.513488] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3500.301527] LustreError: 251164:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3504.853564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3504.880930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3504.937022] Lustre: lustre-OST0001: new disk, initializing [ 3504.938703] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3507.083755] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3510.392142] LustreError: 251164:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3510.399330] LustreError: 251164:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3510.471468] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3511.044315] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3513.855931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3513.859715] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3513.886345] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3520.488096] LustreError: 251164:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3524.075880] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3524.076466] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.082216] LustreError: Skipped 1 previous similar message [ 3524.086792] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3524.092654] Lustre: Skipped 1 previous similar message [ 3527.283099] LustreError: 254653:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3527.285297] LustreError: 254653:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3527.310435] Lustre: server umount lustre-OST0001 complete [ 3529.192795] LustreError: 252993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3529.197918] LustreError: 252993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3534.312168] LustreError: 252993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3534.317083] LustreError: 252993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3540.450473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3540.450524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3540.455609] LustreError: Skipped 1 previous similar message [ 3540.460167] Lustre: Skipped 1 previous similar message [ 3540.461132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3540.471309] Lustre: Skipped 2 previous similar messages [ 3544.550375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3544.551234] LustreError: 252890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3544.552381] Lustre: Skipped 1 previous similar message [ 3544.557913] LustreError: 252890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3549.671084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3549.672815] Lustre: Skipped 1 previous similar message [ 3554.272121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3554.293953] LustreError: 254876:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3554.295621] LustreError: 254876:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3554.328387] Lustre: server umount lustre-OST0000 complete [ 3556.320873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3556.323395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3556.326792] Lustre: Skipped 1 previous similar message [ 3559.905121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3559.907100] Lustre: Skipped 2 previous similar messages [ 3561.626640] Lustre: server umount lustre-MDT0000 complete [ 3562.746092] LustreError: 251150:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879608 with bad export cookie 15885426855649915597 [ 3562.749372] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3562.749671] LustreError: 251150:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3577.312136] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3577.350887] LustreError: 255283:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3577.352907] LustreError: 255283:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3577.408130] Lustre: server umount lustre-MDT0001 complete [ 3581.181867] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 3582.258524] Key type lgssc unregistered [ 3582.386327] LNet: 255816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3582.388781] LNetError: 255816:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3582.395355] LNet: Removed LNI 192.168.204.127@tcp [ 3582.695980] Key type .llcrypt unregistered [ 3582.697069] Key type ._llcrypt unregistered [ 3594.231305] Key type ._llcrypt registered [ 3594.232643] Key type .llcrypt registered [ 3594.278946] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 3600.668647] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 3601.199938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3601.207497] alg: No test for adler32 (adler32-zlib) [ 3602.051125] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 3602.132615] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 3603.712137] Key type lgssc registered [ 3604.076846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3607.597509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3609.907952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3612.055561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3614.256770] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3616.696330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3616.727480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3616.734942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3617.840138] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3617.852789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3617.892307] Lustre: lustre-MDT0000: new disk, initializing [ 3617.922538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3617.928968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3619.234566] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3624.014100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3624.045923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3624.097377] Lustre: 259740:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3624.132015] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3624.135261] Lustre: Skipped 1 previous similar message [ 3624.182691] Lustre: lustre-MDT0001: new disk, initializing [ 3624.225354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3624.241600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3624.244475] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3625.663377] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3628.727470] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3630.061407] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3632.307850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3632.335825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3632.454516] Lustre: lustre-OST0000: new disk, initializing [ 3632.456565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3632.491399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.487218] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3637.975787] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3639.206441] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3639.800536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3639.804388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3639.830961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3640.283175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3641.877285] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3641.980949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3644.903326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3644.905885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3644.906654] LustreError: Skipped 1 previous similar message [ 3644.914286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3644.917725] Lustre: Skipped 1 previous similar message [ 3648.880222] LustreError: 262687:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3648.922922] Lustre: server umount lustre-OST0000 complete [ 3655.138327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3655.138503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3655.139246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3655.142386] Lustre: Skipped 2 previous similar messages [ 3656.776871] LustreError: 262888:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3656.780662] LustreError: 262888:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3656.813558] Lustre: server umount lustre-MDT0000 complete [ 3658.426269] LustreError: 259731:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879704 with bad export cookie 9970035313133912798 [ 3658.429439] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3658.431443] LustreError: 259731:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3658.502291] LustreError: 263091:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3658.504167] LustreError: 263091:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3658.558717] Lustre: server umount lustre-MDT0001 complete [ 3661.941233] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 01:35:07 (1763879707) [ 3664.152562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3664.367618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3665.728821] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3668.886542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3669.011359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3670.558606] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3672.554740] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3673.675483] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3676.436207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3678.860657] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3680.753690] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3729.889435] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3729.893892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3729.901837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3729.905341] Lustre: Skipped 1 previous similar message [ 3735.285218] LustreError: 266102:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3735.289776] LustreError: 266102:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3735.367706] Lustre: server umount lustre-OST0000 complete [ 3737.214383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3741.794205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3741.942051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3741.943782] Lustre: Skipped 1 previous similar message [ 3743.915982] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3745.739357] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3751.148670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3751.152637] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3755.237128] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3755.240617] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3781.093065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3781.094562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3781.094597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3781.094603] LustreError: Skipped 1 previous similar message [ 3781.097211] Lustre: Skipped 2 previous similar messages [ 3781.098959] Lustre: Skipped 2 previous similar messages [ 3785.519190] LustreError: 267764:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3785.523053] LustreError: 267764:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3785.601527] Lustre: server umount lustre-OST0000 complete [ 3791.329749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3791.330251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3791.332893] Lustre: Skipped 1 previous similar message [ 3791.334733] Lustre: Skipped 1 previous similar message [ 3793.047483] Lustre: server umount lustre-MDT0000 complete [ 3794.242188] LustreError: 263742:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879840 with bad export cookie 9970035313133913673 [ 3794.245356] LustreError: 263742:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3794.246194] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3794.307665] LustreError: 268166:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3794.309351] LustreError: 268166:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3794.353939] Lustre: server umount lustre-MDT0001 complete [ 3798.023602] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 3799.000711] Key type lgssc unregistered [ 3799.117305] LNet: 268698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3799.119377] LNetError: 268698:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3799.130297] LNet: Removed LNI 192.168.204.127@tcp [ 3799.390820] Key type .llcrypt unregistered [ 3799.391948] Key type ._llcrypt unregistered [ 3809.586071] Key type ._llcrypt registered [ 3809.587190] Key type .llcrypt registered [ 3809.618608] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 3809.998026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3810.007504] alg: No test for adler32 (adler32-zlib) [ 3810.848124] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 3810.923688] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 3812.496119] Key type lgssc registered [ 3812.810486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3815.066700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3815.072859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3816.221801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3817.364928] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3819.912980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3820.043920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3821.206142] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3822.833993] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3823.885743] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3825.926505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3826.046128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3827.112239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3827.778613] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3829.444755] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3839.460917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3839.462161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3839.462970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3839.465899] Lustre: Skipped 1 previous similar message [ 3839.468381] Lustre: Skipped 1 previous similar message [ 3844.015162] LustreError: 273090:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3844.050979] Lustre: server umount lustre-OST0000 complete [ 3845.600842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3845.603144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3845.607656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3849.697776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3849.698845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3851.397313] LustreError: 273292:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3851.401182] LustreError: 273292:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3851.432034] Lustre: server umount lustre-MDT0000 complete [ 3852.673395] LustreError: 270054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879898 with bad export cookie 7367981538070255135 [ 3852.676147] LustreError: 270054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3852.676629] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3852.746252] LustreError: 273494:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3852.748065] LustreError: 273494:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3852.795133] Lustre: server umount lustre-MDT0001 complete [ 3856.153868] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 3857.363218] Key type lgssc unregistered [ 3857.503314] LNet: 274025:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3857.505426] LNetError: 274025:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3857.514326] LNet: Removed LNI 192.168.204.127@tcp [ 3857.847549] Key type .llcrypt unregistered [ 3857.848663] Key type ._llcrypt unregistered [ 3864.900853] Key type ._llcrypt registered [ 3864.901876] Key type .llcrypt registered [ 3864.947173] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 3865.439430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3865.448334] alg: No test for adler32 (adler32-zlib) [ 3866.315296] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 3866.413683] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 3868.008105] Key type lgssc registered [ 3868.392179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3870.977126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3870.984554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3872.151449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3873.371380] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3876.027430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3876.163968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3877.566255] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3879.328667] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3880.611027] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3883.083813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3883.198706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3885.053535] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3887.014855] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3896.554981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3901.923859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3901.926086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3901.927863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3901.930894] Lustre: Skipped 1 previous similar message [ 3903.279101] LustreError: 277616:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3903.312047] Lustre: server umount lustre-OST0000 complete [ 3906.493329] LustreError: 277817:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3906.495308] LustreError: 277817:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3906.522772] Lustre: server umount lustre-MDT0000 complete [ 3907.041853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3907.042315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3907.042479] LustreError: 276821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3907.042485] LustreError: 276821:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3907.044975] Lustre: Skipped 2 previous similar messages [ 3907.592729] LustreError: 275183:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879953 with bad export cookie 12643765642875468983 [ 3907.595385] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3907.596050] LustreError: 275183:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3907.657266] LustreError: 278019:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3907.658941] LustreError: 278019:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3907.703380] Lustre: server umount lustre-MDT0001 complete [ 3910.964484] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 3912.005881] Key type lgssc unregistered [ 3912.137633] LNet: 278549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3912.141218] LNetError: 278549:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3912.148483] LNet: Removed LNI 192.168.204.127@tcp [ 3912.481583] Key type .llcrypt unregistered [ 3912.483717] Key type ._llcrypt unregistered [ 3922.383501] Key type ._llcrypt registered [ 3922.384964] Key type .llcrypt registered [ 3922.430476] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 3922.895532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3922.908319] alg: No test for adler32 (adler32-zlib) [ 3923.774879] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 3923.856679] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 3925.440128] Key type lgssc registered [ 3925.947880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3929.113608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3929.118911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.299502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3931.894695] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3934.925487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3935.058482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3936.615140] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3938.845181] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3940.126729] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3942.669296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3942.796451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3944.754699] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3946.759313] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3956.142663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3965.408809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3965.411250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3965.415802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3966.436367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3966.436654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3966.438784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3970.799137] LustreError: 282965:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3970.846088] Lustre: server umount lustre-OST0000 complete [ 3976.160949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3976.163334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3976.167550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3976.169331] Lustre: Skipped 1 previous similar message [ 3981.793016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3981.794910] Lustre: Skipped 2 previous similar messages [ 3986.400079] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3986.435802] LustreError: 283166:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3986.437744] LustreError: 283166:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3986.475301] Lustre: server umount lustre-MDT0000 complete [ 3986.913703] LustreError: 281180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3986.918378] LustreError: 281180:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3987.781079] LustreError: 279907:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763880033 with bad export cookie 2060582445528192016 [ 3987.784691] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3987.787825] LustreError: 279907:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3987.857131] LustreError: 283369:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3987.862625] LustreError: 283369:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3987.914783] Lustre: server umount lustre-MDT0001 complete [ 3991.313509] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 3992.630745] Key type lgssc unregistered [ 3992.784574] LNet: 283900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3992.788418] LNetError: 283900:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3992.800656] LNet: Removed LNI 192.168.204.127@tcp [ 3993.139393] Key type .llcrypt unregistered [ 3993.140427] Key type ._llcrypt unregistered [ 4001.007235] Key type ._llcrypt registered [ 4001.008243] Key type .llcrypt registered [ 4001.050183] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4001.567165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4001.597504] alg: No test for adler32 (adler32-zlib) [ 4002.468460] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 4002.552159] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 4004.136131] Key type lgssc registered [ 4004.425642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4007.257862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4007.268505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4008.454464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4009.691129] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4012.935870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4013.122300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4014.402199] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4016.414248] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4017.928000] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4020.436954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4020.621295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4022.824225] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4024.658221] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4034.991799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4040.167589] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4040.168346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4040.169139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4040.174682] Lustre: Skipped 1 previous similar message [ 4041.459267] LustreError: 287527:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4041.505122] Lustre: server umount lustre-OST0000 complete [ 4043.745548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4043.751131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4043.760682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4043.764560] Lustre: Skipped 1 previous similar message [ 4045.282140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4045.283883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4045.391791] LustreError: 287729:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4045.395214] LustreError: 287729:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4045.429887] Lustre: server umount lustre-MDT0000 complete [ 4047.212284] LustreError: 285059:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763880093 with bad export cookie 5992320149394988449 [ 4047.218952] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4047.219030] LustreError: 285059:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4047.317986] LustreError: 287929:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4047.321970] LustreError: 287929:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4047.388452] Lustre: server umount lustre-MDT0001 complete [ 4051.203331] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 4052.575741] Key type lgssc unregistered [ 4052.768726] LNet: 288461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4052.773541] LNetError: 288461:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4052.782751] LNet: Removed LNI 192.168.204.127@tcp [ 4053.236450] Key type .llcrypt unregistered [ 4053.239331] Key type ._llcrypt unregistered [ 4057.123420] Key type ._llcrypt registered [ 4057.125686] Key type .llcrypt registered [ 4057.168878] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4063.023313] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 4069.038186] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4069.384918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4069.419321] alg: No test for adler32 (adler32-zlib) [ 4070.293941] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 4070.370179] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 4071.944095] Key type lgssc registered [ 4072.333672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4075.401475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4078.079596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4080.421855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4083.061254] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4085.779040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4085.804767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4085.812106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4086.930259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4086.946284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4087.002713] Lustre: lustre-MDT0000: new disk, initializing [ 4087.030501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4087.037400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4088.368758] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4093.500285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4093.536954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4093.582449] Lustre: 291945:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4093.606244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4093.607919] Lustre: Skipped 1 previous similar message [ 4093.648728] Lustre: lustre-MDT0001: new disk, initializing [ 4093.691675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4093.711330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4093.716547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4094.894160] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4098.483562] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4099.668294] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4102.212289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4102.248848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4102.392217] Lustre: lustre-OST0000: new disk, initializing [ 4102.394229] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4102.439177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4104.338751] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4107.538453] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4108.799112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4108.801847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4108.843425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4109.063935] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4109.139764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4110.597564] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4110.676546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4113.894017] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4113.894109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4113.897367] LustreError: Skipped 1 previous similar message [ 4113.901156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4113.903978] Lustre: Skipped 1 previous similar message [ 4117.363114] LustreError: 294886:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4117.414441] Lustre: server umount lustre-OST0000 complete [ 4119.011548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4119.012396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4119.017592] Lustre: Skipped 2 previous similar messages [ 4119.022725] Lustre: Skipped 1 previous similar message [ 4124.130344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4124.133796] Lustre: Skipped 1 previous similar message [ 4129.250039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4129.251868] Lustre: Skipped 1 previous similar message [ 4133.344115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4133.348631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4133.350453] Lustre: Skipped 1 previous similar message [ 4133.383864] LustreError: 295087:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4133.385822] LustreError: 295087:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4133.433246] Lustre: server umount lustre-MDT0000 complete [ 4134.947106] LustreError: 291936:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763880180 with bad export cookie 2169588612741002373 [ 4134.953622] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4134.953621] LustreError: 291936:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4135.046480] LustreError: 295290:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4135.048441] LustreError: 295290:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4135.108638] Lustre: server umount lustre-MDT0001 complete [ 4138.429960] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 01:43:04 (1763880184) [ 4140.972033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4141.163409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4142.748982] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4146.008328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4146.129840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4147.575564] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4149.523183] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4150.727754] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4153.095060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4155.512576] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4157.514720] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4171.748336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4171.749723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4171.751541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4171.758195] Lustre: Skipped 1 previous similar message [ 4176.880104] LustreError: 298232:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4176.882145] LustreError: 298232:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4176.918313] Lustre: server umount lustre-OST0000 complete [ 4180.962512] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4180.964586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4184.389304] LustreError: 298435:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4184.393171] LustreError: 298435:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4184.426601] Lustre: server umount lustre-MDT0000 complete [ 4185.891403] LustreError: 295940:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763880231 with bad export cookie 2169588612741003241 [ 4185.894405] LustreError: 295940:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4185.894872] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4186.018173] Lustre: server umount lustre-MDT0001 complete [ 4190.254409] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 4191.385546] Key type lgssc unregistered [ 4191.512620] LNet: 299166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4191.516621] LNetError: 299166:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4191.525653] LNet: Removed LNI 192.168.204.127@tcp [ 4191.884412] Key type .llcrypt unregistered [ 4191.885502] Key type ._llcrypt unregistered [ 4200.790147] Key type ._llcrypt registered [ 4200.791082] Key type .llcrypt registered [ 4200.822588] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 4207.118501] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4207.455050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4207.487670] alg: No test for adler32 (adler32-zlib) [ 4208.323337] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 4208.397908] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 4209.968103] Key type lgssc registered [ 4210.253858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4212.707204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4214.550220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4216.303701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4218.347400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4220.187934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4221.959669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4224.124871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4224.146366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4224.151332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.234679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4225.245488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4225.282491] Lustre: lustre-MDT0000: new disk, initializing [ 4225.307084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4225.313146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4226.444568] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4230.688352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4230.709970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4230.739977] Lustre: 302945:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4230.766422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4230.768092] Lustre: Skipped 1 previous similar message [ 4230.806319] Lustre: lustre-MDT0001: new disk, initializing [ 4230.839565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4230.852325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4230.855757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4232.069398] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4235.415797] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4236.600403] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4238.850665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4238.877189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4239.000543] Lustre: lustre-OST2710: new disk, initializing [ 4239.002384] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4239.037740] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4239.933234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4239.936341] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4239.986371] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4241.086954] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4244.336229] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4246.822729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4246.857632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4246.927327] Lustre: lustre-OST03e8: new disk, initializing [ 4246.929960] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4246.975436] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4247.820295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4247.826180] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4247.857924] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4249.150647] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4250.968051] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4253.552552] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4253.616921] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4254.866907] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4254.924879] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4256.081140] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4256.139256] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4257.368324] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4257.430251] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4261.856769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4261.859091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4261.863188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4263.395181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.395812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4263.398904] Lustre: Skipped 1 previous similar message [ 4263.401849] Lustre: Skipped 3 previous similar messages [ 4264.513685] LustreError: 307288:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4264.544588] Lustre: server umount lustre-MDT0000 complete [ 4265.822480] LustreError: 303724:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763880311 with bad export cookie 15278595893568203232 [ 4265.825477] LustreError: 303724:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4265.826378] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4265.889058] LustreError: 307489:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4265.890730] LustreError: 307489:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4265.942183] Lustre: server umount lustre-MDT0001 complete [ 4277.746134] LustreError: 307689:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4277.747702] LustreError: 307689:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4277.762610] Lustre: server umount lustre-OST2710 complete [ 4289.073376] LustreError: 307891:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4289.075296] LustreError: 307891:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4289.115204] Lustre: server umount lustre-OST03e8 complete [ 4294.626523] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 4297.510638] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4300.570975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4302.715408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4304.977052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4307.318157] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4309.686428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4309.707051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4309.788549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4309.799321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4309.841540] Lustre: lustre-MDT0000: new disk, initializing [ 4309.866303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4309.872552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4311.096237] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4315.246239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4315.267414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4315.295460] Lustre: 310682:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4315.391959] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4316.540781] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4319.758758] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4320.868758] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4323.080635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4323.101546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4323.191740] Lustre: lustre-OST0000: new disk, initializing [ 4323.192978] Lustre: Skipped 1 previous similar message [ 4323.194512] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4323.196519] Lustre: Skipped 2 previous similar messages [ 4323.224374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4323.226227] Lustre: Skipped 1 previous similar message [ 4323.517856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4323.519943] Lustre: Skipped 1 previous similar message [ 4323.522298] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4323.556243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4324.896592] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4328.116735] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4329.671776] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4329.760886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4331.164553] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4331.231994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4334.050478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4334.051409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4334.052827] LustreError: Skipped 1 previous similar message [ 4334.056386] Lustre: Skipped 1 previous similar message [ 4334.058486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4334.062794] Lustre: Skipped 1 previous similar message [ 4339.173823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4339.175711] Lustre: Skipped 1 previous similar message [ 4344.291127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4344.292751] Lustre: Skipped 1 previous similar message [ 4346.336116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4346.354092] LustreError: 313618:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4346.357216] LustreError: 313618:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4346.393953] Lustre: server umount lustre-OST0000 complete [ 4349.409619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.413179] Lustre: Skipped 1 previous similar message [ 4353.815719] Lustre: server umount lustre-MDT0000 complete [ 4355.290190] LustreError: 311461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763880401 with bad export cookie 15278595893568204898 [ 4355.294013] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4355.295734] LustreError: 311461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4355.371951] LustreError: 314023:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4355.374887] LustreError: 314023:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4355.432875] Lustre: server umount lustre-MDT0001 complete [ 4358.234329] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 01:46:44 (1763880404) [ 4358.767772] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4359.370881] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 01:46:45 (1763880405) [ 4360.385250] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4363.208540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4364.552289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4365.725662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4366.978667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4367.409604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4369.995657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4370.099525] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4370.108157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4370.168805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4371.496428] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4374.410079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4374.453308] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4374.462544] Lustre: 316150:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4375.720483] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4377.470391] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4378.991880] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4381.362387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4381.388069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4381.453061] Lustre: Denying initial registration attempt from nid 192.168.204.127@tcp, specified as failover [ 4381.455668] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4381.458225] LustreError: 317766:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 4381.460149] LustreError: 317766:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4381.462041] LustreError: 317766:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4381.484435] Lustre: server umount lustre-OST0000 complete [ 4381.485773] LustreError: 317766:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4384.736944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4384.739092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4384.742816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4384.744427] Lustre: Skipped 2 previous similar messages [ 4389.504569] LustreError: 318133:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4389.506846] LustreError: 318133:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4389.531272] Lustre: server umount lustre-MDT0000 complete [ 4390.659468] LustreError: 316142:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763880436 with bad export cookie 15278595893568205752 [ 4390.662649] LustreError: 316142:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4390.664516] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4390.782147] Lustre: server umount lustre-MDT0001 complete [ 4394.308824] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 4395.649941] Key type lgssc unregistered [ 4395.805331] LNet: 318869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4395.807436] LNetError: 318869:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4395.819394] LNet: Removed LNI 192.168.204.127@tcp [ 4396.138838] Key type .llcrypt unregistered [ 4396.140068] Key type ._llcrypt unregistered [ 4400.062314] Key type ._llcrypt registered [ 4400.063732] Key type .llcrypt registered [ 4400.110986] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4400.469219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4400.494917] alg: No test for adler32 (adler32-zlib) [ 4401.342456] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 4401.434206] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 4403.008152] Key type lgssc registered [ 4403.372675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4405.953762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4407.319261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4408.784242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4410.387380] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4410.813968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4417.230605] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4421.003100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4421.008831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4422.112848] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4422.121565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4422.185320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4423.471628] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4426.779711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4426.815570] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4426.830514] Lustre: 321599:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4426.951811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4428.196429] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4429.789971] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4430.920589] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4433.255680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4433.291072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4433.374908] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4433.494476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4435.788963] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4437.875272] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4438.773249] LustreError: 323875:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4438.827471] Lustre: server umount lustre-OST0000 complete [ 4442.593118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4442.595059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4442.599171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4445.154454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4445.154921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.528631] LustreError: 324084:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 4446.530255] LustreError: 324084:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4446.557373] Lustre: server umount lustre-MDT0000 complete [ 4448.116328] LustreError: 321589:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763880493 with bad export cookie 5695415229467654373 [ 4448.122046] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4448.122822] LustreError: 321589:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4448.219387] LustreError: 324286:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4448.222020] LustreError: 324286:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4448.282097] Lustre: server umount lustre-MDT0001 complete [ 4452.495783] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 4453.744568] Key type lgssc unregistered [ 4453.889546] LNet: 324817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4453.894393] LNetError: 324817:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4453.903730] LNet: Removed LNI 192.168.204.127@tcp [ 4454.324797] Key type .llcrypt unregistered [ 4454.326889] Key type ._llcrypt unregistered [ 4464.135244] Key type ._llcrypt registered [ 4464.136604] Key type .llcrypt registered [ 4464.180130] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4464.471524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4464.528459] alg: No test for adler32 (adler32-zlib) [ 4465.391346] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 4465.473316] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 4467.056127] Key type lgssc registered [ 4467.569988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4470.856567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4470.867631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4472.064931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4473.549955] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4476.582485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4476.701509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4478.299486] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4480.507843] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4481.742626] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4484.713636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4484.893091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4486.894332] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4489.245624] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4502.497735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4502.500556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4502.505220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4504.802752] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.27@tcp (stopping) [ 4506.593612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4506.595794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4507.845163] LustreError: 328463:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4507.882438] Lustre: server umount lustre-MDT0000 complete [ 4509.078621] LustreError: 326173:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763880554 with bad export cookie 13967054985933468403 [ 4509.084211] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4509.084463] LustreError: 326173:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4509.916635] LustreError: 326189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4509.923302] LustreError: 326189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4509.925063] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.27@tcp (stopping) [ 4509.928198] Lustre: Skipped 2 previous similar messages [ 4511.713261] LustreError: 326189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4511.713484] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4511.725247] Lustre: Skipped 1 previous similar message [ 4514.721148] LustreError: 326189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4514.721406] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4514.726380] Lustre: Skipped 1 previous similar message [ 4515.204344] LustreError: 328665:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4515.206027] LustreError: 328665:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4515.257902] Lustre: server umount lustre-MDT0001 complete [ 4517.268615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4521.044082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4533.216639] LustreError: 325501:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008de00deb x1849562770727040/t0(0) o250->MGC192.168.204.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4533.323335] LustreError: 327801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4533.328930] LustreError: 327801:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4533.345587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4533.366620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4534.437595] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4536.932316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4538.166925] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4539.111237] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4539.795244] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4543.207718] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4545.381410] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4546.444423] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4546.959459] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4548.580608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4548.581236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4548.582465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4548.582473] Lustre: Skipped 1 previous similar message [ 4548.583713] Lustre: Skipped 1 previous similar message [ 4562.400150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4562.423293] LustreError: 330969:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4562.424887] LustreError: 330969:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4562.459086] Lustre: server umount lustre-OST0000 complete [ 4563.938550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4563.945558] Lustre: Skipped 1 previous similar message [ 4569.057953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4569.060114] Lustre: Skipped 8 previous similar messages [ 4578.272280] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4578.322942] LustreError: 331180:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4578.324905] LustreError: 331180:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4578.359354] Lustre: server umount lustre-MDT0000 complete [ 4579.297810] LustreError: 329478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4579.304790] LustreError: 329478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4579.766893] LustreError: 326910:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763880625 with bad export cookie 13967054985933475557 [ 4579.771797] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4579.772071] LustreError: 326910:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4579.917181] Lustre: server umount lustre-MDT0001 complete [ 4583.928169] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 4585.064747] Key type lgssc unregistered [ 4585.211363] LNet: 331914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4585.214335] LNetError: 331914:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4585.222345] LNet: Removed LNI 192.168.204.127@tcp [ 4585.536609] Key type .llcrypt unregistered [ 4585.538869] Key type ._llcrypt unregistered [ 4594.075227] Key type ._llcrypt registered [ 4594.076099] Key type .llcrypt registered [ 4594.117138] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4594.485044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4594.533123] alg: No test for adler32 (adler32-zlib) [ 4595.397194] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 4595.486083] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 4597.072132] Key type lgssc registered [ 4597.465524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4599.940216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4599.945638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4601.111705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4602.312409] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4605.014592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4605.143933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4606.452781] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4608.180467] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4609.365241] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4610.529755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4610.532400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4610.537556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4614.625934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4614.626971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4616.445526] LustreError: 334677:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4616.474743] Lustre: server umount lustre-MDT0000 complete [ 4617.684889] LustreError: 334679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763880663 with bad export cookie 1764158842600761580 [ 4617.687724] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4617.687806] LustreError: 334679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4617.755034] LustreError: 334880:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4617.756931] LustreError: 334880:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4617.813160] Lustre: server umount lustre-MDT0001 complete [ 4622.336588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4622.436525] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4622.448052] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4622.524455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4623.742564] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4626.441060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4626.472171] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4626.482625] Lustre: 335649:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4626.586379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4627.858018] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4629.638548] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4630.791761] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4633.282624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4633.349711] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4633.353071] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4633.355345] LustreError: 337305:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4633.357320] LustreError: 337305:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4633.359046] LustreError: 337305:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4633.381082] Lustre: server umount lustre-OST0000 complete [ 4633.382253] LustreError: 337305:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4636.098893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4636.171764] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4636.251186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4637.522629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4638.025629] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4639.634329] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4642.129664] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4642.146554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4642.171708] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4642.200450] Lustre: lustre-OST0001: new disk, initializing [ 4642.202255] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4643.870757] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4645.544105] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4646.429075] LustreError: 339412:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4646.430779] LustreError: 339412:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4646.445980] Lustre: server umount lustre-OST0001 complete [ 4647.418536] LustreError: 337770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4647.423394] LustreError: 337770:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4647.905044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4647.905220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4647.906921] LustreError: Skipped 1 previous similar message [ 4647.910309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4647.912840] Lustre: Skipped 2 previous similar messages [ 4652.516165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4652.517035] LustreError: 337770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.519223] Lustre: Skipped 1 previous similar message [ 4652.523026] LustreError: 337770:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4653.677083] LustreError: 339622:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4653.678631] LustreError: 339622:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4653.711866] Lustre: server umount lustre-OST0000 complete [ 4657.633636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4657.633940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4657.638156] Lustre: Skipped 1 previous similar message [ 4657.639413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4657.640670] Lustre: Skipped 1 previous similar message [ 4660.863748] LustreError: 339824:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup [ 4660.865373] LustreError: 339824:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4660.890811] Lustre: server umount lustre-MDT0000 complete [ 4662.035135] LustreError: 335639:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763880707 with bad export cookie 1764158842600762105 [ 4662.038033] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4662.040300] LustreError: 335639:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4662.144898] Lustre: server umount lustre-MDT0001 complete [ 4665.346900] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 4666.282901] Key type lgssc unregistered [ 4666.413438] LNet: 340557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4666.417515] LNetError: 340557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4666.425307] LNet: Removed LNI 192.168.204.127@tcp [ 4666.680812] Key type .llcrypt unregistered [ 4666.681801] Key type ._llcrypt unregistered [ 4677.875152] Key type ._llcrypt registered [ 4677.876775] Key type .llcrypt registered [ 4677.913866] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 4685.113164] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4685.447614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4685.463063] alg: No test for adler32 (adler32-zlib) [ 4686.311698] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 4686.389740] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 4687.968238] Key type lgssc registered [ 4688.453390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4692.134894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4694.893146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4697.295190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4699.202566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4701.336050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4701.358101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4701.364103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4702.457422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4702.469653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4702.506680] Lustre: lustre-MDT0000: new disk, initializing [ 4702.533410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4702.539196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4703.696399] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4707.674508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4707.706312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4707.757951] Lustre: 344490:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4707.798751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4707.802467] Lustre: Skipped 1 previous similar message [ 4707.861926] Lustre: lustre-MDT0001: new disk, initializing [ 4707.916416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4707.943204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4707.947649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4709.583773] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4712.869133] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4713.977421] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4716.007226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4716.027297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4716.127918] Lustre: lustre-OST0000: new disk, initializing [ 4716.129613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4716.157944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4717.829085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4717.831840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4717.880400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4718.029420] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4720.991356] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4722.230780] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4722.301680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4723.519928] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4723.591157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4728.292898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4728.295377] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4728.297868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4728.299705] LustreError: Skipped 1 previous similar message [ 4730.287111] LustreError: 347427:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4730.324469] Lustre: server umount lustre-OST0000 complete [ 4733.411386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4733.411559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4733.412082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4733.412086] Lustre: Skipped 1 previous similar message [ 4733.414713] Lustre: Skipped 2 previous similar messages [ 4737.676513] LustreError: 347629:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4737.681045] LustreError: 347629:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4737.721811] Lustre: server umount lustre-MDT0000 complete [ 4739.000462] LustreError: 347631:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763880784 with bad export cookie 9938095146405127706 [ 4739.004417] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4739.068092] LustreError: 347832:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4739.069754] LustreError: 347832:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4739.123058] Lustre: server umount lustre-MDT0001 complete [ 4741.822178] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 01:53:07 (1763880787) [ 4743.579944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4745.331206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4751.904732] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 4755.094890] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4758.132135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4760.205421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4762.130973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4764.090907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4766.242240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4766.262075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4766.347344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4766.356973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4766.395135] Lustre: lustre-MDT0000: new disk, initializing [ 4766.416852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4766.422594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4767.552065] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4771.353652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4771.372237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4771.401030] Lustre: 351599:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4771.420524] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4771.421969] Lustre: Skipped 1 previous similar message [ 4771.457083] Lustre: lustre-MDT0001: new disk, initializing [ 4771.486392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4771.496431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4771.499242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4772.611313] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4775.574698] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4776.766451] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4778.859342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4778.884953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4779.406548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4779.442254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4780.740576] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4783.671514] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4784.933621] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4784.993227] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4786.093256] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4786.152481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4788.783657] LustreError: 354536:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4788.785316] LustreError: 354536:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4788.826341] Lustre: server umount lustre-OST0000 complete [ 4792.288949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4792.292708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4792.299146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4792.300669] Lustre: Skipped 1 previous similar message [ 4794.849061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4794.849993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4799.969256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4799.970887] Lustre: Skipped 2 previous similar messages [ 4799.972830] LustreError: 354737:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4799.974450] LustreError: 354737:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4799.997558] Lustre: server umount lustre-MDT0000 complete [ 4801.072133] LustreError: 351591:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763880846 with bad export cookie 9938095146405128574 [ 4801.074870] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4801.075120] LustreError: 351591:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4810.212449] LustreError: 354939:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4810.215252] LustreError: 354939:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4810.255336] Lustre: server umount lustre-MDT0001 complete [ 4812.866500] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 01:54:18 (1763880858) [ 4815.077336] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 01:54:21 (1763880861) [ 4817.419596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4817.573914] LustreError: 355953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4817.578462] LustreError: 355953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4817.596387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4817.598050] Lustre: Skipped 1 previous similar message [ 4818.772271] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4821.353958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4822.624375] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4824.280164] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4825.434363] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4827.539359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4827.617888] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4829.282513] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4830.870287] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4840.230300] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4840.757455] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4841.239199] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4841.751101] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4842.330329] Lustre: Failing over lustre-MDT0000 [ 4842.475980] LustreError: 358393:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4842.478541] LustreError: 358393:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4842.503763] Lustre: server umount lustre-MDT0000 complete [ 4847.074657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4847.074866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4847.075607] LustreError: 358106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4847.077472] Lustre: Skipped 2 previous similar messages [ 4852.193019] LustreError: 355971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4852.195805] LustreError: 355971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4855.230225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4855.267663] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4855.367788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4855.369685] Lustre: Skipped 2 previous similar messages [ 4855.384815] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4856.608175] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4857.564967] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4860.388366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4860.395228] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4860.408778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4861.397701] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4861.854906] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4863.604410] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4864.178959] Lustre: Failing over lustre-MDT0000 [ 4864.337401] Lustre: server umount lustre-MDT0000 complete [ 4865.506570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4865.506574] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4865.507757] LustreError: 355947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4865.515013] LustreError: 355947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4870.625765] LustreError: 355953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4870.629807] LustreError: 355953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4876.945584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4876.982469] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4877.104226] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4878.045457] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4878.266314] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4882.409819] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4882.411293] Lustre: Skipped 2 previous similar messages [ 4882.416887] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4882.433421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4883.443755] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4883.912366] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4887.525042] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4887.525394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4887.526030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4887.528211] Lustre: Skipped 3 previous similar messages [ 4892.155090] LustreError: 360496:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4892.156719] LustreError: 360496:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 4892.183887] Lustre: server umount lustre-MDT0000 complete [ 4892.640959] LustreError: 355953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4892.644826] LustreError: 355953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4893.260415] LustreError: 357570:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763880939 with bad export cookie 9938095146405131241 [ 4893.262894] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4893.263329] LustreError: 357570:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4893.366088] Lustre: server umount lustre-MDT0001 complete [ 4904.660183] Lustre: server umount lustre-OST0000 complete [ 4909.282543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4909.435521] LustreError: 361731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4909.455338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4909.456990] Lustre: Skipped 1 previous similar message [ 4910.539036] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4913.200752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4914.563093] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4916.474412] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4917.982737] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4920.259459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4922.084935] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4924.032935] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4934.693996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4938.720703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4938.722669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4938.725485] Lustre: Skipped 1 previous similar message [ 4938.726972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4938.728631] Lustre: Skipped 2 previous similar messages [ 4942.089670] Lustre: server umount lustre-OST0000 complete [ 4949.268272] Lustre: server umount lustre-MDT0000 complete [ 4949.984902] LustreError: 361731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4949.989219] LustreError: 361731:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4950.362141] LustreError: 361710:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763880996 with bad export cookie 9938095146405131934 [ 4950.365314] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4950.365635] LustreError: 361710:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4950.470411] Lustre: server umount lustre-MDT0001 complete [ 4954.060055] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 4954.906942] Key type lgssc unregistered [ 4955.018366] LNet: 364932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4955.020489] LNetError: 364932:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4955.029297] LNet: Removed LNI 192.168.204.127@tcp [ 4955.279922] Key type .llcrypt unregistered [ 4955.281024] Key type ._llcrypt unregistered [ 4964.076215] Key type ._llcrypt registered [ 4964.077317] Key type .llcrypt registered [ 4964.107840] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4964.467085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4964.483288] alg: No test for adler32 (adler32-zlib) [ 4965.320758] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 4965.393570] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 4966.968126] Key type lgssc registered [ 4967.259247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4969.492090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4969.497236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4970.644475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4971.719012] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4974.185430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4974.305207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4975.464175] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4977.067238] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4978.084289] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4980.051445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4980.160231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4981.222771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 4981.754092] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4983.262322] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4989.410237] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4989.411567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4989.413098] Lustre: Skipped 2 previous similar messages [ 4989.415435] Lustre: Skipped 2 previous similar messages [ 4993.848897] LustreError: 368650:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4993.870311] Lustre: server umount lustre-MDT0000 complete [ 4994.529304] LustreError: 366401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4994.532721] LustreError: 366401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4994.848815] LustreError: 366386:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763881040 with bad export cookie 13344429609493679194 [ 4994.850987] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4994.851300] LustreError: 366386:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4994.907953] LustreError: 368852:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4994.909427] LustreError: 368852:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4994.943734] Lustre: server umount lustre-MDT0001 complete [ 5005.169267] LustreError: 369053:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5005.170781] LustreError: 369053:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5005.200917] Lustre: server umount lustre-OST0000 complete [ 5008.787174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5008.915072] LustreError: 369781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5008.933338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5009.894027] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5012.183573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5013.363631] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5014.863243] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5015.909549] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5017.777230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5017.865428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5017.867075] Lustre: Skipped 1 previous similar message [ 5018.927952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5019.696311] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5021.556978] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5028.323654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5028.325979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5028.326768] Lustre: Skipped 2 previous similar messages [ 5028.328222] Lustre: Skipped 1 previous similar message [ 5033.443354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5033.447187] Lustre: Skipped 2 previous similar messages [ 5034.383119] LustreError: 372126:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5034.386965] LustreError: 372126:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5034.421871] Lustre: server umount lustre-MDT0000 complete [ 5036.003467] LustreError: 369760:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763881081 with bad export cookie 13344429609493680293 [ 5036.007224] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5036.007611] LustreError: 369760:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5036.136171] Lustre: server umount lustre-MDT0001 complete [ 5047.857042] LustreError: 372530:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5047.858833] LustreError: 372530:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5047.898631] Lustre: server umount lustre-OST0000 complete [ 5052.236189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5052.371672] LustreError: 373358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5052.391414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5053.442964] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5055.932728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5057.165528] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5058.736624] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5059.816742] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5061.844385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5063.019965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5063.627474] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5065.200953] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5068.256697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5068.258634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5068.262107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5071.330220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5071.332252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5072.302772] LustreError: 375597:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5072.304387] LustreError: 375597:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5072.338705] Lustre: server umount lustre-OST0000 complete [ 5076.451114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5076.451483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5076.457436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5076.458944] Lustre: Skipped 3 previous similar messages [ 5079.640979] Lustre: server umount lustre-MDT0000 complete [ 5080.769204] LustreError: 373384:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763881126 with bad export cookie 13344429609493681301 [ 5080.773473] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5080.881185] Lustre: server umount lustre-MDT0001 complete [ 5084.302275] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 5085.239340] Key type lgssc unregistered [ 5085.362326] LNet: 376538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5085.364426] LNetError: 376538:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5085.370451] LNet: Removed LNI 192.168.204.127@tcp [ 5085.658705] Key type .llcrypt unregistered [ 5085.659680] Key type ._llcrypt unregistered [ 5094.098635] Key type ._llcrypt registered [ 5094.099597] Key type .llcrypt registered [ 5094.133912] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5094.462536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5094.502458] alg: No test for adler32 (adler32-zlib) [ 5095.325357] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 5095.385859] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 5096.952116] Key type lgssc registered [ 5097.200801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5099.148839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5099.153049] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5099.154966] LustreError: 377936:0:(osd_handler.c:8577:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5099.163773] LustreError: 377936:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5099.165407] LustreError: 377936:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5099.166978] LustreError: 377936:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5099.168828] LustreError: 377936:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5101.217128] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5101.219280] LustreError: 378333:0:(osd_handler.c:8577:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5101.226840] LustreError: 378333:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5101.228764] LustreError: 378333:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5101.230721] LustreError: 378333:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5101.232832] LustreError: 378333:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5106.578786] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 5107.440964] Key type lgssc unregistered [ 5107.556406] LNet: 379316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5107.560063] LNetError: 379316:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5107.567295] LNet: Removed LNI 192.168.204.127@tcp [ 5107.822440] Key type .llcrypt unregistered [ 5107.823431] Key type ._llcrypt unregistered [ 5112.937496] Key type ._llcrypt registered [ 5112.938415] Key type .llcrypt registered [ 5112.970126] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 5118.525110] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5118.930351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5118.954064] alg: No test for adler32 (adler32-zlib) [ 5119.797914] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 5119.868244] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 5121.440075] Key type lgssc registered [ 5121.729161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5124.226430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5126.137819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5128.054883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5130.043897] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5132.085753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5132.105018] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5132.109493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5133.190430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5133.202465] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5133.238891] Lustre: lustre-MDT0000: new disk, initializing [ 5133.264986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5133.274158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5134.451140] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5138.287690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5138.305964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5138.335429] Lustre: 382419:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5138.356209] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5138.358087] Lustre: Skipped 1 previous similar message [ 5138.392731] Lustre: lustre-MDT0001: new disk, initializing [ 5138.423216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5138.434093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5138.437137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5139.528783] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5142.284173] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5143.279861] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5145.210888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5145.234533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5145.322283] Lustre: lustre-OST0000: new disk, initializing [ 5145.324206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5145.350588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5146.929153] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5148.598911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5148.601598] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5148.634714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5149.670086] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5150.775327] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5150.833333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5151.978625] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5152.072891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5153.764387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5153.764663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5153.767869] LustreError: Skipped 1 previous similar message [ 5153.771572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5153.775896] Lustre: Skipped 1 previous similar message [ 5154.734687] LustreError: 385356:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5154.774392] Lustre: server umount lustre-OST0000 complete [ 5158.882716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5158.882878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5158.890698] Lustre: Skipped 1 previous similar message [ 5158.893788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5158.896550] Lustre: Skipped 1 previous similar message [ 5162.176546] LustreError: 385558:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5162.179891] LustreError: 385558:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5162.212486] Lustre: server umount lustre-MDT0000 complete [ 5163.481783] LustreError: 382411:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763881209 with bad export cookie 7380933229922417304 [ 5163.486977] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5163.566103] LustreError: 385760:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5163.567746] LustreError: 385760:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5163.618414] Lustre: server umount lustre-MDT0001 complete [ 5166.086329] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5166.687175] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 02:00:12 (1763881212) [ 5168.857337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5169.043920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5170.186308] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5172.630138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5173.830230] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5175.366465] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5176.438912] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5178.391569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5178.489778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5178.493055] Lustre: Skipped 1 previous similar message [ 5180.082198] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5181.637494] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5183.637251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5183.659600] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5183.706864] Lustre: lustre-OST0001: new disk, initializing [ 5183.708437] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5184.959682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5184.962626] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5185.000172] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5185.393642] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5188.275182] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5190.116880] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5190.117401] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5190.118761] LustreError: Skipped 1 previous similar message [ 5190.121651] Lustre: Skipped 1 previous similar message [ 5190.123157] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5190.125197] Lustre: Skipped 1 previous similar message [ 5194.469342] Lustre: lustre-OST0001: Not available for connect from 192.168.204.27@tcp (stopping) [ 5196.077210] LustreError: 389678:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5196.078819] LustreError: 389678:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5196.096169] Lustre: server umount lustre-OST0001 complete [ 5200.355324] LustreError: 388075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5200.359019] LustreError: 388075:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5205.477959] LustreError: 388076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5205.482305] LustreError: 388076:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5208.545743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5208.548210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5208.552193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5208.553974] Lustre: Skipped 2 previous similar messages [ 5210.594790] LustreError: 388284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5210.599145] LustreError: 388284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5215.716288] LustreError: 388075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5215.720863] LustreError: 388075:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5220.834851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5220.835740] LustreError: 388284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5220.836280] Lustre: Skipped 5 previous similar messages [ 5220.839431] LustreError: 388284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5222.368073] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5222.382442] LustreError: 389900:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5222.383937] LustreError: 389900:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5222.409959] Lustre: server umount lustre-OST0000 complete [ 5223.904697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5223.906482] LustreError: Skipped 1 previous similar message [ 5223.907583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5223.910273] Lustre: Skipped 1 previous similar message [ 5229.647416] Lustre: server umount lustre-MDT0000 complete [ 5230.613922] LustreError: 386458:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763881276 with bad export cookie 7380933229922418172 [ 5230.616402] LustreError: 386458:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5230.616585] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5236.732938] LustreError: 390316:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5236.734374] LustreError: 390316:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5236.773536] Lustre: server umount lustre-MDT0001 complete [ 5239.409472] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 5240.176918] Key type lgssc unregistered [ 5240.296309] LNet: 390848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5240.298302] LNetError: 390848:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5240.305292] LNet: Removed LNI 192.168.204.127@tcp [ 5240.560886] Key type .llcrypt unregistered [ 5240.561852] Key type ._llcrypt unregistered [ 5250.361146] Key type ._llcrypt registered [ 5250.362075] Key type .llcrypt registered [ 5250.396319] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 5255.953709] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5256.447622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5256.457492] alg: No test for adler32 (adler32-zlib) [ 5257.298605] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 5257.368287] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 5258.936076] Key type lgssc registered [ 5259.256140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5261.607105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5263.304547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5264.960776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5266.830802] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5268.918606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5268.938289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5268.944386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5270.033220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5270.043748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5270.079482] Lustre: lustre-MDT0000: new disk, initializing [ 5270.104157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5270.109838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5271.258830] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5275.201964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5275.226512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5275.256617] Lustre: 394768:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5275.281823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5275.283319] Lustre: Skipped 1 previous similar message [ 5275.321086] Lustre: lustre-MDT0001: new disk, initializing [ 5275.355335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5275.366250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5275.369230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5276.599186] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5279.495325] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5280.552403] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5282.595775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5282.630801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5282.739387] Lustre: lustre-OST0000: new disk, initializing [ 5282.741191] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5282.773637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5283.395518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5283.398236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5283.446258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5284.478248] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5287.411031] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5288.666966] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5288.748472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5289.961650] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5290.036684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5293.537172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5293.539111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5293.542585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5294.053991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5294.054862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5294.055799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5296.688109] LustreError: 397705:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5296.721797] Lustre: server umount lustre-OST0000 complete [ 5299.171197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5299.171807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5299.174378] Lustre: Skipped 1 previous similar message [ 5299.177395] Lustre: Skipped 2 previous similar messages [ 5304.064348] LustreError: 397906:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5304.066106] LustreError: 397906:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5304.093776] Lustre: server umount lustre-MDT0000 complete [ 5305.237162] LustreError: 395547:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763881351 with bad export cookie 4189383026402144429 [ 5305.242234] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5305.302838] LustreError: 398108:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5305.304605] LustreError: 398108:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5305.356461] Lustre: server umount lustre-MDT0001 complete [ 5308.081470] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 02:02:34 (1763881354) [ 5310.925708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5313.462188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5313.612121] Lustre: lustre-MDT0000: new disk, initializing [ 5313.622807] LustreError: 399373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5313.627235] LustreError: 399373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5313.647739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5314.793406] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5317.264213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5318.468199] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5319.987890] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5321.056440] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5321.674092] LustreError: 399402:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5321.676411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5321.676839] LustreError: 399402:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5321.677085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5321.679603] Lustre: Skipped 1 previous similar message [ 5321.684294] LustreError: 399402:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 5326.817071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5326.819238] Lustre: Skipped 2 previous similar messages [ 5327.804325] LustreError: 400713:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5327.806824] LustreError: 400713:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5327.830307] Lustre: server umount lustre-MDT0000 complete [ 5327.841289] LustreError: 399373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5327.848301] LustreError: 399373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5329.043061] LustreError: 400715:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763881374 with bad export cookie 4189383026402145297 [ 5329.046601] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5329.049269] LustreError: 400715:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5329.147828] Lustre: server umount lustre-MDT0001 complete [ 5332.082824] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 02:02:58 (1763881378) [ 5334.201313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5334.356753] LustreError: 401635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5334.380068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5334.382185] Lustre: Skipped 1 previous similar message [ 5335.535848] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5338.168658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5339.483564] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5341.163600] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5342.207707] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5344.451644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5344.590478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5344.592508] Lustre: Skipped 1 previous similar message [ 5346.316667] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5348.156242] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5351.767770] Lustre: Permanently deactivating lustre-OST0000 [ 5351.772309] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5358.582363] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5360.040034] LustreError: 404377:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5360.043013] LustreError: 404377:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 5360.044773] LustreError: 404377:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5360.574677] LustreError: 404425:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 5360.576689] LustreError: 404425:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5367.215090] LustreError: 404523:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5367.217114] LustreError: 404523:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5367.247828] Lustre: server umount lustre-OST0000 complete [ 5368.035405] LustreError: 404676:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 5368.037712] LustreError: 404676:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5369.312720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5369.315097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5369.318948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5378.531444] LustreError: 404773:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5378.533483] LustreError: 404773:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5378.554040] Lustre: server umount lustre-MDT0000 complete [ 5379.744142] LustreError: 403983:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763881425 with bad export cookie 4189383026402145829 [ 5379.746902] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5379.747674] LustreError: 403983:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5389.322663] Lustre: server umount lustre-MDT0001 complete [ 5391.831829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5392.908168] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5393.730884] LustreError: 405817:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5395.961984] Lustre: 406155:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.127@tcp [ 5396.749837] LustreError: 406253:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5396.753287] LustreError: 406253:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5396.780363] Lustre: server umount MGS complete [ 5400.543273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5400.709399] LustreError: 406880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5400.715102] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5400.716975] Lustre: Skipped 1 previous similar message [ 5400.741028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5402.097026] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5405.005216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5405.139358] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5406.613829] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5408.761202] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5410.223263] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5411.003815] Lustre: Permanently reactivating lustre-OST0000 [ 5411.011641] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5411.015622] Lustre: Skipped 2 previous similar messages [ 5438.944128] LustreError: 408532:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5442.035533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5442.146313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5442.148449] Lustre: Skipped 1 previous similar message [ 5443.828385] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5445.683567] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5458.915826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5458.917066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5458.917833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5458.917840] Lustre: Skipped 4 previous similar messages [ 5458.918541] LustreError: Skipped 1 previous similar message [ 5458.921989] Lustre: Skipped 2 previous similar messages [ 5459.308743] LustreError: 410357:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5459.310354] LustreError: 410357:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5459.343918] Lustre: server umount lustre-OST0000 complete [ 5462.607931] Lustre: server umount lustre-MDT0000 complete [ 5463.704176] LustreError: 410564:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763881509 with bad export cookie 4189383026402146774 [ 5463.707942] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5463.708155] LustreError: 410564:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5463.806511] Lustre: server umount lustre-MDT0001 complete [ 5467.208548] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 5468.037607] Key type lgssc unregistered [ 5468.158340] LNet: 411296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5468.160584] LNetError: 411296:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5468.171289] LNet: Removed LNI 192.168.204.127@tcp [ 5468.412930] Key type .llcrypt unregistered [ 5468.414337] Key type ._llcrypt unregistered [ 5473.958859] Key type ._llcrypt registered [ 5473.960009] Key type .llcrypt registered [ 5473.990759] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 5480.152642] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5480.435687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5480.445131] alg: No test for adler32 (adler32-zlib) [ 5481.285086] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 5481.359499] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 5482.936111] Key type lgssc registered [ 5483.247029] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5485.912047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5487.769487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5489.500521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5491.266703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5493.263363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5493.280181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5493.284560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5494.363692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5494.373391] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5494.407901] Lustre: lustre-MDT0000: new disk, initializing [ 5494.430291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5494.436185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5495.536107] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5499.259707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5499.283991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5499.313355] Lustre: 414402:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5499.333492] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5499.334957] Lustre: Skipped 1 previous similar message [ 5499.368570] Lustre: lustre-MDT0001: new disk, initializing [ 5499.399880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5499.410083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5499.412832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5500.531459] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5503.389897] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5504.492347] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5506.550504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5506.573912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5506.670406] Lustre: lustre-OST0000: new disk, initializing [ 5506.672061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5506.695646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5508.269228] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5511.140143] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5512.301841] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5513.718368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5513.721548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5513.747243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5514.392106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5515.498978] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5515.558440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5518.817953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5518.820127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5518.821138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5518.828096] Lustre: Skipped 1 previous similar message [ 5522.158080] LustreError: 417356:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5522.192137] Lustre: server umount lustre-OST0000 complete [ 5523.937268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5523.937797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5523.940397] Lustre: Skipped 1 previous similar message [ 5523.941944] Lustre: Skipped 1 previous similar message [ 5529.057228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5529.058792] Lustre: Skipped 1 previous similar message [ 5529.406467] LustreError: 417557:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5529.408173] LustreError: 417557:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5529.436166] Lustre: server umount lustre-MDT0000 complete [ 5530.519047] LustreError: 414393:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763881576 with bad export cookie 12061386044625406753 [ 5530.522074] LustreError: 414393:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5530.522613] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5530.594113] LustreError: 417759:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5530.595796] LustreError: 417759:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5530.645651] Lustre: server umount lustre-MDT0001 complete [ 5533.110803] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5533.593514] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 02:06:19 (1763881579) [ 5535.686820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5535.865239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5537.065935] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5539.730356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5541.059472] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5542.684669] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5543.776365] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5545.875445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5545.955273] LustreError: 420073:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5545.959243] LustreError: 420073:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5545.999136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5546.000973] Lustre: Skipped 1 previous similar message [ 5547.765689] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5549.315939] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5549.912603] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5551.362884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5555.174597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5555.175657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5555.178808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5555.179623] LustreError: Skipped 2 previous similar messages [ 5555.182243] Lustre: Skipped 1 previous similar message [ 5558.571047] LustreError: 420912:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5558.572638] LustreError: 420912:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5558.598392] Lustre: server umount lustre-OST0000 complete [ 5560.289539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5560.289756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5560.290213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5560.295156] Lustre: Skipped 2 previous similar messages [ 5565.752995] LustreError: 421113:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5565.754479] LustreError: 421113:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5565.777821] Lustre: server umount lustre-MDT0000 complete [ 5566.808832] LustreError: 418457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763881612 with bad export cookie 12061386044625407621 [ 5566.811256] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5566.811381] LustreError: 418457:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5566.912719] Lustre: server umount lustre-MDT0001 complete [ 5570.342992] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 5571.200704] Key type lgssc unregistered [ 5571.325352] LNet: 421846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5571.329119] LNetError: 421846:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5571.337447] LNet: Removed LNI 192.168.204.127@tcp [ 5571.628988] Key type .llcrypt unregistered [ 5571.630924] Key type ._llcrypt unregistered [ 5578.880655] Key type ._llcrypt registered [ 5578.881595] Key type .llcrypt registered [ 5578.912602] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 5578.995311] Key type .llcrypt unregistered [ 5578.996230] Key type ._llcrypt unregistered [ 5585.685788] Key type ._llcrypt registered [ 5585.687455] Key type .llcrypt registered [ 5585.719887] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5586.035131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5586.084459] alg: No test for adler32 (adler32-zlib) [ 5586.918704] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 5586.990282] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 5588.560108] Key type lgssc registered [ 5588.886833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5591.141699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5591.146838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5592.300679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5593.460133] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5595.942638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5596.043615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5597.638226] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5599.174188] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5601.152478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5601.278370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5602.351475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5602.441508] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5606.369810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5606.371848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5606.375233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5607.396474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5607.398535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5607.399082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5607.402864] Lustre: Skipped 1 previous similar message [ 5611.183104] LustreError: 426049:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5611.213911] Lustre: server umount lustre-OST0000 complete [ 5612.513552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.516995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5612.518429] Lustre: Skipped 1 previous similar message [ 5617.633392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5617.634821] Lustre: Skipped 1 previous similar message [ 5618.371923] LustreError: 426251:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5618.375078] LustreError: 426251:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5618.404162] Lustre: server umount lustre-MDT0000 complete [ 5619.456108] LustreError: 423976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763881665 with bad export cookie 18009229240946812729 [ 5619.461484] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5619.461710] LustreError: 423976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5619.526937] LustreError: 426452:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5619.528568] LustreError: 426452:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5619.575624] Lustre: server umount lustre-MDT0001 complete [ 5622.722945] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 5623.652353] Key type lgssc unregistered [ 5623.768433] LNet: 426982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5623.770595] LNetError: 426982:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5623.778475] LNet: Removed LNI 192.168.204.127@tcp [ 5624.062041] Key type .llcrypt unregistered [ 5624.062898] Key type ._llcrypt unregistered [ 5632.600874] Key type ._llcrypt registered [ 5632.601834] Key type .llcrypt registered [ 5632.636936] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5632.940465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5632.954440] alg: No test for adler32 (adler32-zlib) [ 5633.802442] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 5633.881939] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 5635.456095] Key type lgssc registered [ 5635.810093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5638.279382] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5638.284832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5653.728250] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5658.848113] LustreError: 428329:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.127@tcp: failed processing log, type 4: rc = -110 [ 5685.472110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5687.166340] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5688.823155] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5690.866704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5691.045364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5691.072599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5692.216154] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5694.866534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5694.994229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5696.152053] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5697.782429] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5698.956829] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5701.600711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5701.602773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5701.606284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5705.185782] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5705.187817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.190717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5706.212196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5706.213678] Lustre: Skipped 1 previous similar message [ 5707.182090] LustreError: 430609:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 5707.210251] Lustre: server umount lustre-OST0000 complete [ 5710.304755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5710.306487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5710.309321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5714.364265] LustreError: 430811:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5714.366088] LustreError: 430811:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5714.391637] Lustre: server umount lustre-MDT0000 complete [ 5715.522513] LustreError: 428337:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763881761 with bad export cookie 8620366255869935344 [ 5715.525247] LustreError: 428337:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5715.525676] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5715.582399] LustreError: 431015:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5715.584155] LustreError: 431015:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5715.631293] Lustre: server umount lustre-MDT0001 complete [ 5718.830349] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 5719.702959] Key type lgssc unregistered [ 5719.813352] LNet: 431546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5719.815340] LNetError: 431546:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5719.821400] LNet: Removed LNI 192.168.204.127@tcp [ 5720.116768] Key type .llcrypt unregistered [ 5720.118492] Key type ._llcrypt unregistered [ 5728.258059] Key type ._llcrypt registered [ 5728.258936] Key type .llcrypt registered [ 5728.295979] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5728.800242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5728.808779] alg: No test for adler32 (adler32-zlib) [ 5729.659479] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 5729.734767] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 5731.312094] Key type lgssc registered [ 5731.655294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5734.056030] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5734.061376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5735.216676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5736.409565] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5739.098644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5739.230404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5740.438830] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5742.050149] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5743.117850] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5745.224705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5745.343548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5746.412182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5747.004992] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5748.528610] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5754.853210] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5754.854623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5754.854870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5754.855886] Lustre: Skipped 2 previous similar messages [ 5754.859704] Lustre: Skipped 2 previous similar messages [ 5756.348640] LustreError: 435162:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5756.376327] Lustre: server umount lustre-MDT0000 complete [ 5756.991287] LustreError: 432900:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.27@tcp arrived at 1763881802 with bad export cookie 4827564292015527445 [ 5756.997270] LustreError: 432900:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5757.567233] LustreError: 432900:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763881803 with bad export cookie 4827564292015526591 [ 5757.571606] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5759.969582] LustreError: 433630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5759.969838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5759.970706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.971693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5759.971699] Lustre: Skipped 1 previous similar message [ 5759.973316] LustreError: 433630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5762.976684] LustreError: 433630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5762.977196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5762.979989] LustreError: 433630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5763.629083] LustreError: 435370:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5763.630665] LustreError: 435370:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5763.661256] Lustre: server umount lustre-OST0000 complete [ 5765.504591] LustreError: 435723:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5765.506156] LustreError: 435723:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5765.553245] Lustre: server umount lustre-MDT0001 complete [ 5768.206778] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 5769.062299] Key type lgssc unregistered [ 5769.181436] LNet: 436253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5769.183457] LNetError: 436253:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5769.192443] LNet: Removed LNI 192.168.204.127@tcp [ 5769.470031] Key type .llcrypt unregistered [ 5769.470935] Key type ._llcrypt unregistered [ 5777.187785] Key type ._llcrypt registered [ 5777.188681] Key type .llcrypt registered [ 5777.220630] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5777.593298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5777.648975] alg: No test for adler32 (adler32-zlib) [ 5778.482764] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 5778.552697] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 5780.128149] Key type lgssc registered [ 5780.428742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5782.574263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5782.579230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5783.715934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5784.792269] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5787.161065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5787.280690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5788.361752] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5789.778082] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5790.694698] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5792.712610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5792.825282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5793.834068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5794.527525] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5796.078590] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5797.857747] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5797.859881] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5797.863492] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5797.897967] LustreError: 439869:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5797.918595] Lustre: server umount lustre-MDT0001 complete [ 5798.883310] LustreError: 438351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5798.884843] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5798.887373] LustreError: 438351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5804.003337] LustreError: 438351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5804.004206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5804.004609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5804.005296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5805.422040] LustreError: 440073:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5805.424689] LustreError: 440073:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5805.457799] Lustre: server umount lustre-OST0000 complete [ 5806.608890] LustreError: 440275:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5806.611518] LustreError: 440275:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5806.670444] Lustre: server umount lustre-MDT0000 complete [ 5810.440763] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 5811.420034] Key type lgssc unregistered [ 5811.543448] LNet: 440958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5811.545632] LNetError: 440958:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5811.552399] LNet: Removed LNI 192.168.204.127@tcp [ 5811.845429] Key type .llcrypt unregistered [ 5811.847064] Key type ._llcrypt unregistered [ 5819.030863] Key type ._llcrypt registered [ 5819.031802] Key type .llcrypt registered [ 5819.065762] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 5824.585774] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5824.944451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5825.022870] alg: No test for adler32 (adler32-zlib) [ 5825.883905] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 5825.966122] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 5827.544120] Key type lgssc registered [ 5827.905980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5830.594286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5832.573152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5834.528988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5836.442189] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5839.436413] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5842.413578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5844.518996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5844.540801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5844.544772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5845.636309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5845.647075] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5845.687448] Lustre: lustre-MDT0000: new disk, initializing [ 5845.714748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5845.720642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5846.961376] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5850.236063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5852.332367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5852.360596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5852.456921] Lustre: lustre-OST0000: new disk, initializing [ 5852.458729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5852.460235] Lustre: Skipped 1 previous similar message [ 5852.481790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5853.317138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5853.319737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5853.327718] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5854.170242] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5857.170838] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5860.289203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5862.413686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5862.439141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5862.470420] Lustre: 444903:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5862.502897] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5862.540547] Lustre: lustre-MDT0001: new disk, initializing [ 5862.587837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5862.603977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 5862.606971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 5863.826905] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5866.962310] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5867.038081] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5868.121107] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5872.250366] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 4 sec [ 5874.916023] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.27@tcp (stopping) [ 5877.220172] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5877.220180] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5877.222289] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5877.224991] Lustre: Skipped 1 previous similar message [ 5879.677018] LustreError: 448353:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5879.704219] Lustre: server umount lustre-MDT0001 complete [ 5887.458690] LustreError: 448087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5887.462151] LustreError: 448087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5892.579139] LustreError: 448087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5892.579469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5892.587865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5892.589037] Lustre: Skipped 1 previous similar message [ 5897.698506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5897.699267] LustreError: 448087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5902.818525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5902.819267] LustreError: 448087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5905.888130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5905.905073] LustreError: 448626:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 5905.906427] LustreError: 448626:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5905.937386] Lustre: server umount lustre-OST0000 complete [ 5913.210735] LustreError: 448828:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5913.212185] LustreError: 448828:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5913.266054] Lustre: server umount lustre-MDT0000 complete [ 5917.066992] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 5917.965246] Key type lgssc unregistered [ 5918.081520] LNet: 449511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5918.083243] LNetError: 449511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5918.092460] LNet: Removed LNI 192.168.204.127@tcp [ 5918.357074] Key type .llcrypt unregistered [ 5918.357887] Key type ._llcrypt unregistered [ 5929.289369] Key type ._llcrypt registered [ 5929.290234] Key type .llcrypt registered [ 5929.330228] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 5929.423176] Key type .llcrypt unregistered [ 5929.424462] Key type ._llcrypt unregistered [ 5935.325834] Key type ._llcrypt registered [ 5935.326690] Key type .llcrypt registered [ 5935.361512] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5935.727184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5935.733655] alg: No test for adler32 (adler32-zlib) [ 5936.589172] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 5936.677082] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 5938.256113] Key type lgssc registered [ 5938.603201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5940.832886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5942.636164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5944.399595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5946.337116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5946.356103] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5946.359847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5947.439386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5947.449565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5947.487634] Lustre: lustre-MDT0000: new disk, initializing [ 5947.512989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5947.519612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5948.654502] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5952.356779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5952.383406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5952.413132] Lustre: 452834:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5952.436078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5952.437566] Lustre: Skipped 1 previous similar message [ 5952.474869] Lustre: lustre-MDT0001: new disk, initializing [ 5952.507926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5952.518864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5952.523489] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5953.735223] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5956.594559] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5957.671284] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5959.676637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5959.709546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5959.808918] Lustre: lustre-OST0000: new disk, initializing [ 5959.810683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5959.835729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5960.441126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5960.443948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5960.497844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5961.524145] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5964.371915] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5967.841086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5967.843082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5967.846487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5970.916297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5970.916790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5970.917932] Lustre: Skipped 1 previous similar message [ 5970.922653] Lustre: Skipped 1 previous similar message [ 5972.670039] LustreError: 455273:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5972.697140] Lustre: server umount lustre-MDT0000 complete [ 5973.795089] LustreError: 452824:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763882019 with bad export cookie 11168511862588601181 [ 5973.798110] LustreError: 452824:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5973.798167] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5973.861259] LustreError: 455476:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5973.862950] LustreError: 455476:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5973.904922] Lustre: server umount lustre-MDT0001 complete [ 5985.137529] LustreError: 455676:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5985.140753] LustreError: 455676:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5985.177016] Lustre: server umount lustre-OST0000 complete [ 5987.978098] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 02:13:54 (1763882034) [ 5988.267319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5990.124028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5990.262787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5991.308201] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5993.675727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5994.877572] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5996.312470] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5997.298826] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5999.142069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5999.164336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5999.230644] Lustre: Found index 0 for lustre-OST0000, updating log [ 5999.285853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5999.287290] Lustre: Skipped 1 previous similar message [ 6000.295165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6000.791473] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6002.222664] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6004.194224] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6004.195537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6004.198891] Lustre: Skipped 2 previous similar messages [ 6006.753501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6006.754862] Lustre: Skipped 2 previous similar messages [ 6009.263081] LustreError: 458750:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6009.264500] LustreError: 458750:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6009.290597] Lustre: server umount lustre-OST0000 complete [ 6010.053205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6014.432687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6014.434439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6014.437342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6014.438637] Lustre: Skipped 2 previous similar messages [ 6016.696956] LustreError: 459013:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6016.698554] LustreError: 459013:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6016.718206] Lustre: server umount lustre-MDT0000 complete [ 6017.707973] LustreError: 456484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763882063 with bad export cookie 11168511862588602560 [ 6017.710430] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6017.710797] LustreError: 456484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6017.803301] Lustre: server umount lustre-MDT0001 complete [ 6020.152853] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 02:14:26 (1763882066) [ 6020.448063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6020.709871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6020.971972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6021.230859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6021.492704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6021.773203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6022.060252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6022.342181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6022.622982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6022.905081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6023.191076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6023.474693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6023.762146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6024.049151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6024.328324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6024.611076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6024.888190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6025.167170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6025.444184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6025.731106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6026.010926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6026.295176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6026.575036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6026.863159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6027.148185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6027.432394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6027.718084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6028.002191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6028.288042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6028.569153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6028.850745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6029.141183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6029.426317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6029.713248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6029.997058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6030.289316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6030.565983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6030.857378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6031.145156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6031.426049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6031.713139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6032.001280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6032.285062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6032.564916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6032.854118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6033.144109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6033.432101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6033.720114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6034.007298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6034.299034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6034.579173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6036.782147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6036.919109] LustreError: 462594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6036.923746] LustreError: 462594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6036.940227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6038.008279] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6040.465280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6041.668214] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6043.239562] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6044.308153] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6046.270256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6046.296990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6046.379057] Lustre: Found index 0 for lustre-OST0000, updating log [ 6048.037456] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6048.552095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6049.628620] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6051.297738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6051.300916] Lustre: Skipped 1 previous similar message [ 6051.302540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6051.304107] Lustre: Skipped 1 previous similar message [ 6056.298329] LustreError: 464888:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6056.299933] LustreError: 464888:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6056.325918] Lustre: server umount lustre-OST0000 complete [ 6059.552234] LNet: 1 local NIs in recovery (showing 1): 192.168.204.127@tcp [ 6061.024628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6061.026471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6063.438421] Lustre: server umount lustre-MDT0000 complete [ 6064.446880] LustreError: 465100:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763882110 with bad export cookie 11168511862588603617 [ 6064.449261] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6064.449603] LustreError: 465100:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6064.540486] Lustre: server umount lustre-MDT0001 complete [ 6065.297944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6069.486222] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 6072.043300] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 6074.554569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6076.271182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6077.932326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6079.592283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6081.340264] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 02:15:27 (1763882127) [ 6135.277828] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 6136.000367] Key type lgssc unregistered [ 6136.101393] LNet: 468886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6136.103237] LNetError: 468886:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6178.086335] LNet: Removed LNI 192.168.204.127@tcp [ 6178.318669] Key type .llcrypt unregistered [ 6178.319742] Key type ._llcrypt unregistered [ 6184.035831] Key type ._llcrypt registered [ 6184.036634] Key type .llcrypt registered [ 6184.062826] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 6184.284536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6184.293454] alg: No test for adler32 (adler32-zlib) [ 6185.116519] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 6185.177912] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 6186.744078] Key type lgssc registered [ 6186.992914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6188.120970] LNet: Added LNI 192.168.204.127@tcp100 [8/256/0/180] [ 6188.371438] LNet: Added LNI 192.168.204.127@tcp101 [8/256/0/180] [ 6188.622559] LNet: Added LNI 192.168.204.127@tcp102 [8/256/0/180] [ 6188.873885] LNet: Added LNI 192.168.204.127@tcp103 [8/256/0/180] [ 6189.121294] LNet: Added LNI 192.168.204.127@tcp104 [8/256/0/180] [ 6189.372188] LNet: Added LNI 192.168.204.127@tcp105 [8/256/0/180] [ 6189.629709] LNet: Added LNI 192.168.204.127@tcp106 [8/256/0/180] [ 6189.878238] LNet: Added LNI 192.168.204.127@tcp107 [8/256/0/180] [ 6190.124929] LNet: Added LNI 192.168.204.127@tcp108 [8/256/0/180] [ 6190.371193] LNet: Added LNI 192.168.204.127@tcp109 [8/256/0/180] [ 6190.616870] LNet: Added LNI 192.168.204.127@tcp110 [8/256/0/180] [ 6190.868172] LNet: Added LNI 192.168.204.127@tcp111 [8/256/0/180] [ 6191.116042] LNet: Added LNI 192.168.204.127@tcp112 [8/256/0/180] [ 6191.364519] LNet: Added LNI 192.168.204.127@tcp113 [8/256/0/180] [ 6191.613952] LNet: Added LNI 192.168.204.127@tcp114 [8/256/0/180] [ 6191.863657] LNet: Added LNI 192.168.204.127@tcp115 [8/256/0/180] [ 6192.116735] LNet: Added LNI 192.168.204.127@tcp116 [8/256/0/180] [ 6192.368363] LNet: Added LNI 192.168.204.127@tcp117 [8/256/0/180] [ 6192.616401] LNet: Added LNI 192.168.204.127@tcp118 [8/256/0/180] [ 6192.866904] LNet: Added LNI 192.168.204.127@tcp119 [8/256/0/180] [ 6193.115922] LNet: Added LNI 192.168.204.127@tcp120 [8/256/0/180] [ 6193.364494] LNet: Added LNI 192.168.204.127@tcp121 [8/256/0/180] [ 6193.612794] LNet: Added LNI 192.168.204.127@tcp122 [8/256/0/180] [ 6193.862638] LNet: Added LNI 192.168.204.127@tcp123 [8/256/0/180] [ 6194.111760] LNet: Added LNI 192.168.204.127@tcp124 [8/256/0/180] [ 6194.363429] LNet: Added LNI 192.168.204.127@tcp125 [8/256/0/180] [ 6194.609336] LNet: Added LNI 192.168.204.127@tcp126 [8/256/0/180] [ 6194.859090] LNet: Added LNI 192.168.204.127@tcp127 [8/256/0/180] [ 6195.108560] LNet: Added LNI 192.168.204.127@tcp128 [8/256/0/180] [ 6195.357509] LNet: Added LNI 192.168.204.127@tcp129 [8/256/0/180] [ 6195.603184] LNet: Added LNI 192.168.204.127@tcp130 [8/256/0/180] [ 6195.852373] LNet: Added LNI 192.168.204.127@tcp131 [8/256/0/180] [ 6196.099436] LNet: Added LNI 192.168.204.127@tcp132 [8/256/0/180] [ 6196.344806] LNet: Added LNI 192.168.204.127@tcp133 [8/256/0/180] [ 6196.594342] LNet: Added LNI 192.168.204.127@tcp134 [8/256/0/180] [ 6196.844377] LNet: Added LNI 192.168.204.127@tcp135 [8/256/0/180] [ 6198.721448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6198.740042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6198.743729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6199.777410] Lustre: 471859:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.127@tcp: can't add NID 192.168.204.127@tcp130: rc = -75 [ 6199.839374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6199.849251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6199.883336] Lustre: lustre-MDT0000: new disk, initializing [ 6199.910961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6199.916414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6200.984846] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6204.658396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6204.685717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6204.721233] Lustre: 471878:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6204.746375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6204.747835] Lustre: Skipped 1 previous similar message [ 6204.781388] Lustre: lustre-MDT0001: new disk, initializing [ 6204.791921] Lustre: 472663:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.127@tcp: can't add NID 192.168.204.127@tcp130: rc = -75 [ 6204.793730] Lustre: 472663:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6204.831518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6204.842793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6204.845430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6205.973307] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6208.775166] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6209.812372] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6211.752802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6211.782446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6211.884800] Lustre: lustre-OST0000: new disk, initializing [ 6211.886527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6211.918613] Lustre: 473633:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.127@tcp: can't add NID 192.168.204.127@tcp130: rc = -75 [ 6211.920991] Lustre: 473633:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [ 6211.936444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6213.533233] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6216.319531] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6218.287216] LustreError: 474457:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6218.317023] Lustre: server umount lustre-OST0000 complete [ 6219.759053] Lustre: 474659:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.127@tcp: can't add NID 192.168.204.127@tcp130: rc = -75 [ 6219.761402] Lustre: 474659:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6219.770112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6219.771860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6219.772949] Lustre: Skipped 1 previous similar message [ 6224.865693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6224.867092] Lustre: Skipped 2 previous similar messages [ 6225.465095] LustreError: 474657:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6225.467176] LustreError: 474657:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6225.493776] Lustre: server umount lustre-MDT0000 complete [ 6226.503358] LustreError: 472662:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763882272 with bad export cookie 16498986924370950038 [ 6226.508641] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6226.559843] LustreError: 474862:0:(obd_class.h:479:obd_check_dev()) Device 7 not setup [ 6226.561917] LustreError: 474862:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6226.613750] Lustre: server umount lustre-MDT0001 complete [ 6229.400352] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 6230.314671] Key type lgssc unregistered [ 6230.459299] LNet: 475401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6230.461751] LNetError: 475401:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6230.463575] LNet: Removed LNI 192.168.204.127@tcp [ 6231.520105] LNet: Removed LNI 192.168.204.127@tcp100 [ 6231.521121] LNetError: 475401:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6231.522876] LNetError: 475401:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6231.524834] LNet: Removed LNI 192.168.204.127@tcp101 [ 6232.544097] LNet: Removed LNI 192.168.204.127@tcp102 [ 6232.545227] LNetError: 475401:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6232.546929] LNetError: 475401:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6232.548883] LNet: Removed LNI 192.168.204.127@tcp103 [ 6233.568144] LNet: Removed LNI 192.168.204.127@tcp104 [ 6234.592076] LNet: Removed LNI 192.168.204.127@tcp105 [ 6234.593100] LNetError: 475401:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6234.594707] LNetError: 475401:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 6234.596451] LNet: Removed LNI 192.168.204.127@tcp106 [ 6235.616126] LNet: Removed LNI 192.168.204.127@tcp107 [ 6236.640072] LNet: Removed LNI 192.168.204.127@tcp108 [ 6237.664070] LNet: Removed LNI 192.168.204.127@tcp109 [ 6238.688072] LNet: Removed LNI 192.168.204.127@tcp110 [ 6238.689094] LNetError: 475401:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6238.690730] LNetError: 475401:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 6238.692518] LNet: Removed LNI 192.168.204.127@tcp111 [ 6238.693455] LNet: Removed LNI 192.168.204.127@tcp112 [ 6238.694343] LNet: Removed LNI 192.168.204.127@tcp113 [ 6238.695287] LNet: Removed LNI 192.168.204.127@tcp114 [ 6238.696188] LNet: Removed LNI 192.168.204.127@tcp115 [ 6238.697146] LNet: Removed LNI 192.168.204.127@tcp116 [ 6238.698006] LNet: Removed LNI 192.168.204.127@tcp117 [ 6238.698912] LNet: Removed LNI 192.168.204.127@tcp118 [ 6238.699795] LNet: Removed LNI 192.168.204.127@tcp119 [ 6238.700724] LNet: Removed LNI 192.168.204.127@tcp120 [ 6238.701673] LNet: Removed LNI 192.168.204.127@tcp121 [ 6238.702584] LNet: Removed LNI 192.168.204.127@tcp122 [ 6238.703513] LNet: Removed LNI 192.168.204.127@tcp123 [ 6238.704415] LNet: Removed LNI 192.168.204.127@tcp124 [ 6238.705321] LNet: Removed LNI 192.168.204.127@tcp125 [ 6238.706261] LNet: Removed LNI 192.168.204.127@tcp126 [ 6238.707169] LNet: Removed LNI 192.168.204.127@tcp127 [ 6238.708002] LNet: Removed LNI 192.168.204.127@tcp128 [ 6238.708861] LNet: Removed LNI 192.168.204.127@tcp129 [ 6238.709836] LNet: Removed LNI 192.168.204.127@tcp130 [ 6238.710799] LNet: Removed LNI 192.168.204.127@tcp131 [ 6238.711706] LNet: Removed LNI 192.168.204.127@tcp132 [ 6238.712663] LNet: Removed LNI 192.168.204.127@tcp133 [ 6238.713535] LNet: Removed LNI 192.168.204.127@tcp134 [ 6238.720530] LNet: Removed LNI 192.168.204.127@tcp135 [ 6238.942295] Key type .llcrypt unregistered [ 6238.943220] Key type ._llcrypt unregistered [ 6244.859595] Key type ._llcrypt registered [ 6244.860520] Key type .llcrypt registered [ 6244.894066] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 6250.195763] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 6250.592620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6250.633356] alg: No test for adler32 (adler32-zlib) [ 6251.474157] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 6251.548942] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 6253.128105] Key type lgssc registered [ 6253.457537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6255.937655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6257.749389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6259.433512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6261.170303] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6262.969254] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 02:18:29 (1763882309) [ 6267.809450] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 6268.628786] Key type lgssc unregistered [ 6268.742272] LNet: 479412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6268.744212] LNetError: 479412:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6268.753308] LNet: Removed LNI 192.168.204.127@tcp [ 6268.983419] Key type .llcrypt unregistered [ 6268.984370] Key type ._llcrypt unregistered [ 6274.741978] Key type ._llcrypt registered [ 6274.742931] Key type .llcrypt registered [ 6274.774130] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 6275.067680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6275.102514] alg: No test for adler32 (adler32-zlib) [ 6275.940443] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 6276.010980] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 6277.584055] Key type lgssc registered [ 6277.835090] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6280.314373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6280.332151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6280.336155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6281.409163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6281.418757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6281.452756] Lustre: lustre-MDT0000: new disk, initializing [ 6281.474859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6281.480276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6282.572480] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6286.224038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6286.246032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6286.274573] Lustre: 480585:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6286.294909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6286.296469] Lustre: Skipped 1 previous similar message [ 6286.331092] Lustre: lustre-MDT0001: new disk, initializing [ 6286.360431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6286.370794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6286.373597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6287.445630] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6290.154705] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6291.132356] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6293.019313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6293.039460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6293.128627] Lustre: lustre-OST0000: new disk, initializing [ 6293.130326] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6293.154354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6293.217222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6293.219752] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6293.258326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6294.763527] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6297.485433] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6298.400900] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing lnet_if_list [ 6299.719225] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 6299.728493] LNet: Added LNI 192.168.204.127@tcp10 [8/256/0/180] [ 6301.108251] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 6301.116510] LNet: Added LNI 192.168.204.127@tcp11 [8/256/0/180] [ 6302.138394] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 6302.146776] LNet: Added LNI 192.168.204.127@tcp12 [8/256/0/180] [ 6303.199511] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 6303.207784] LNet: Added LNI 192.168.204.127@tcp13 [8/256/0/180] [ 6304.296358] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 6304.304868] LNet: Added LNI 192.168.204.127@tcp14 [8/256/0/180] [ 6305.428436] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 6305.441500] LNet: Added LNI 192.168.204.127@tcp15 [8/256/0/180] [ 6306.637556] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 6306.646241] LNet: Added LNI 192.168.204.127@tcp16 [8/256/0/180] [ 6307.836375] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 6307.844674] LNet: Added LNI 192.168.204.127@tcp17 [8/256/0/180] [ 6309.058436] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 6309.066956] LNet: Added LNI 192.168.204.127@tcp18 [8/256/0/180] [ 6310.304599] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 6310.313268] LNet: Added LNI 192.168.204.127@tcp19 [8/256/0/180] [ 6311.600219] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 6311.608856] LNet: Added LNI 192.168.204.127@tcp20 [8/256/0/180] [ 6312.911236] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 6312.919585] LNet: Added LNI 192.168.204.127@tcp21 [8/256/0/180] [ 6314.265282] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 6314.274320] LNet: Added LNI 192.168.204.127@tcp22 [8/256/0/180] [ 6315.661123] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 6315.669952] LNet: Added LNI 192.168.204.127@tcp23 [8/256/0/180] [ 6317.082117] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 6317.090886] LNet: Added LNI 192.168.204.127@tcp24 [8/256/0/180] [ 6318.543572] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 6318.551948] LNet: Added LNI 192.168.204.127@tcp25 [8/256/0/180] [ 6320.032223] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 6320.041748] LNet: Added LNI 192.168.204.127@tcp26 [8/256/0/180] [ 6321.641198] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 6321.650208] LNet: Added LNI 192.168.204.127@tcp27 [8/256/0/180] [ 6321.658997] LNetError: 479843:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [ 6321.661396] LNetError: Protocol error connecting to 192.168.204.127@tcp26 at host 192.168.204.127:988: is it running a compatible version of Lustre? [ 6323.255493] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 6323.265132] LNet: Added LNI 192.168.204.127@tcp28 [8/256/0/180] [ 6323.269118] LNetError: 479845:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [ 6323.272521] LNetError: Protocol error connecting to 192.168.204.127@tcp26 at host 192.168.204.127:988: is it running a compatible version of Lustre? [ 6323.275899] LNetError: Skipped 1 previous similar message [ 6324.893201] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 6324.902662] LNet: Added LNI 192.168.204.127@tcp29 [8/256/0/180] [ 6324.906199] LNetError: 479843:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [ 6324.909198] LNetError: 479843:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [ 6324.911831] LNetError: Protocol error connecting to 192.168.204.127@tcp27 at host 192.168.204.127:988: is it running a compatible version of Lustre? [ 6324.914677] LNetError: Skipped 1 previous similar message [ 6326.608538] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 6326.618302] LNet: Added LNI 192.168.204.127@tcp30 [8/256/0/180] [ 6327.201340] LNet: 1 local NIs in recovery (showing 1): 192.168.204.127@tcp29 [ 6327.204976] LNetError: 479844:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [ 6327.207080] LNetError: 479844:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 6327.208733] LNetError: Protocol error connecting to 192.168.204.127@tcp29 at host 192.168.204.127:988: is it running a compatible version of Lustre? [ 6327.210860] LNetError: Skipped 2 previous similar messages [ 6328.386057] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 6328.395852] LNet: Added LNI 192.168.204.127@tcp31 [8/256/0/180] [ 6330.135492] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 6330.144786] LNet: Added LNI 192.168.204.127@tcp32 [8/256/0/180] [ 6331.297876] LNetError: 479842:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [ 6331.300421] LNetError: 479842:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 2 previous similar messages [ 6331.302275] LNetError: Protocol error connecting to 192.168.204.127@tcp30 at host 192.168.204.127:988: is it running a compatible version of Lustre? [ 6331.304691] LNetError: Skipped 2 previous similar messages [ 6331.965540] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 6331.974734] LNet: Added LNI 192.168.204.127@tcp33 [8/256/0/180] [ 6333.823667] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 6333.833336] LNet: Added LNI 192.168.204.127@tcp34 [8/256/0/180] [ 6335.688134] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 6335.697704] LNet: Added LNI 192.168.204.127@tcp35 [8/256/0/180] [ 6337.579808] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 6337.588378] LNet: Added LNI 192.168.204.127@tcp36 [8/256/0/180] [ 6339.519171] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 6339.527810] LNet: Added LNI 192.168.204.127@tcp37 [8/256/0/180] [ 6339.531700] LNetError: 479844:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [ 6339.534822] LNetError: 479844:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 13 previous similar messages [ 6339.537063] LNetError: Protocol error connecting to 192.168.204.127@tcp35 at host 192.168.204.127:988: is it running a compatible version of Lustre? [ 6339.539627] LNetError: Skipped 13 previous similar messages [ 6341.534926] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 6341.545578] LNet: Added LNI 192.168.204.127@tcp38 [8/256/0/180] [ 6343.497831] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 6343.506785] LNet: Added LNI 192.168.204.127@tcp39 [8/256/0/180] [ 6345.542257] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 6345.551907] LNet: Added LNI 192.168.204.127@tcp40 [8/256/0/180] [ 6347.589139] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 6347.598416] LNet: Added LNI 192.168.204.127@tcp41 [8/256/0/180] [ 6349.697931] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 6349.706754] LNet: Added LNI 192.168.204.127@tcp42 [8/256/0/180] [ 6351.836296] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 6351.845561] LNet: Added LNI 192.168.204.127@tcp43 [8/256/0/180] [ 6354.001576] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 6354.010530] LNet: Added LNI 192.168.204.127@tcp44 [8/256/0/180] [ 6355.874248] LNetError: 479842:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [ 6355.876878] LNetError: 479842:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 28 previous similar messages [ 6355.878904] LNetError: Protocol error connecting to 192.168.204.127@tcp42 at host 192.168.204.127:988: is it running a compatible version of Lustre? [ 6355.881467] LNetError: Skipped 28 previous similar messages [ 6356.236241] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 6356.245536] LNet: Added LNI 192.168.204.127@tcp45 [8/256/0/180] [ 6358.754682] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 6358.768902] LNet: Added LNI 192.168.204.127@tcp46 [8/256/0/180] [ 6361.189547] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 6361.199202] LNet: Added LNI 192.168.204.127@tcp47 [8/256/0/180] [ 6363.383221] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 6363.391786] LNet: Added LNI 192.168.204.127@tcp48 [8/256/0/180] [ 6365.638986] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 6365.647829] LNet: Added LNI 192.168.204.127@tcp49 [8/256/0/180] [ 6368.085520] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 6368.094446] LNet: Added LNI 192.168.204.127@tcp50 [8/256/0/180] [ 6375.394526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6375.395098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6375.396580] LustreError: Skipped 1 previous similar message [ 6375.399955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6375.401944] Lustre: Skipped 1 previous similar message [ 6380.514295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6380.515738] Lustre: Skipped 1 previous similar message [ 6380.970915] LustreError: 504755:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6380.998967] Lustre: server umount lustre-OST0000 complete [ 6383.584623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6383.586337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6383.588949] Lustre: Skipped 1 previous similar message [ 6383.590267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6385.633122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6385.633588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6387.616116] LNet: 7 local NIs in recovery (showing 7): 192.168.204.127@tcp42, 192.168.204.127@tcp47, 192.168.204.127@tcp39, 192.168.204.127@tcp34, 192.168.204.127@tcp32, 192.168.204.127@tcp30, 192.168.204.127@tcp29 [ 6388.640722] LNetError: 479843:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [ 6388.642945] LNetError: 479843:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 51 previous similar messages [ 6388.644639] LNetError: Protocol error connecting to 192.168.204.127@tcp31 at host 192.168.204.127:988: is it running a compatible version of Lustre? [ 6388.646853] LNetError: Skipped 51 previous similar messages [ 6390.752845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6390.754176] Lustre: Skipped 2 previous similar messages [ 6396.384087] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6396.411350] LustreError: 504961:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6396.412819] LustreError: 504961:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6396.437925] Lustre: server umount lustre-MDT0000 complete [ 6397.451962] LustreError: 480575:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763882443 with bad export cookie 3707757213738772406 [ 6397.454653] LustreError: 480575:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6397.455090] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6397.514372] LustreError: 505164:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6397.515856] LustreError: 505164:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6397.556127] Lustre: server umount lustre-MDT0001 complete [ 6400.427918] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 6401.176052] Key type lgssc unregistered [ 6401.300455] LNet: 505703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6401.302795] LNetError: 505703:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6401.304490] LNet: Removed LNI 192.168.204.127@tcp [ 6402.336108] LNet: Removed LNI 192.168.204.127@tcp10 [ 6402.337116] LNetError: 505703:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6402.338697] LNetError: 505703:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6402.340458] LNet: Removed LNI 192.168.204.127@tcp11 [ 6403.360093] LNet: Removed LNI 192.168.204.127@tcp12 [ 6403.361116] LNetError: 505703:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6403.362850] LNetError: 505703:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6403.364616] LNet: Removed LNI 192.168.204.127@tcp13 [ 6404.384125] LNet: Removed LNI 192.168.204.127@tcp14 [ 6405.408070] LNet: Removed LNI 192.168.204.127@tcp15 [ 6405.409096] LNetError: 505703:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6405.410764] LNetError: 505703:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 6405.412511] LNet: Removed LNI 192.168.204.127@tcp16 [ 6406.432087] LNet: Removed LNI 192.168.204.127@tcp17 [ 6407.456128] LNet: Removed LNI 192.168.204.127@tcp18 [ 6408.480135] LNet: Removed LNI 192.168.204.127@tcp19 [ 6409.504080] LNet: Removed LNI 192.168.204.127@tcp20 [ 6409.505088] LNetError: 505703:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6409.506672] LNetError: 505703:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 6409.508494] LNet: Removed LNI 192.168.204.127@tcp21 [ 6410.528087] LNet: Removed LNI 192.168.204.127@tcp22 [ 6411.552129] LNet: Removed LNI 192.168.204.127@tcp23 [ 6412.576072] LNet: Removed LNI 192.168.204.127@tcp24 [ 6413.600097] LNet: Removed LNI 192.168.204.127@tcp25 [ 6414.624084] LNet: Removed LNI 192.168.204.127@tcp26 [ 6415.648068] LNet: Removed LNI 192.168.204.127@tcp27 [ 6416.672133] LNet: Removed LNI 192.168.204.127@tcp28 [ 6416.673117] LNet: Removed LNI 192.168.204.127@tcp29 [ 6417.696089] LNet: Removed LNI 192.168.204.127@tcp30 [ 6417.697125] LNetError: 505703:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6417.698738] LNetError: 505703:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 6417.700510] LNet: Removed LNI 192.168.204.127@tcp31 [ 6417.701406] LNet: Removed LNI 192.168.204.127@tcp32 [ 6418.720066] LNet: Removed LNI 192.168.204.127@tcp33 [ 6418.721050] LNet: Removed LNI 192.168.204.127@tcp34 [ 6419.744145] LNet: Removed LNI 192.168.204.127@tcp35 [ 6420.768147] LNet: Removed LNI 192.168.204.127@tcp36 [ 6421.792073] LNet: Removed LNI 192.168.204.127@tcp37 [ 6422.816091] LNet: Removed LNI 192.168.204.127@tcp38 [ 6422.817141] LNet: Removed LNI 192.168.204.127@tcp39 [ 6423.840141] LNet: Removed LNI 192.168.204.127@tcp40 [ 6424.864097] LNet: Removed LNI 192.168.204.127@tcp41 [ 6424.865064] LNet: Removed LNI 192.168.204.127@tcp42 [ 6425.888065] LNet: Removed LNI 192.168.204.127@tcp43 [ 6426.912134] LNet: Removed LNI 192.168.204.127@tcp44 [ 6427.936137] LNet: Removed LNI 192.168.204.127@tcp45 [ 6428.960101] LNet: Removed LNI 192.168.204.127@tcp46 [ 6428.961325] LNet: Removed LNI 192.168.204.127@tcp47 [ 6429.984075] LNet: Removed LNI 192.168.204.127@tcp48 [ 6429.984991] LNet: Removed LNI 192.168.204.127@tcp49 [ 6429.991374] LNet: Removed LNI 192.168.204.127@tcp50 [ 6430.206306] Key type .llcrypt unregistered [ 6430.207284] Key type ._llcrypt unregistered [ 6436.930073] Key type ._llcrypt registered [ 6436.931067] Key type .llcrypt registered [ 6436.961386] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 6437.320810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6437.327432] alg: No test for adler32 (adler32-zlib) [ 6438.173866] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 6438.250797] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 6439.824082] Key type lgssc registered [ 6440.151608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6442.459955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6442.465393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6443.616900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6444.764717] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6447.283163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6447.396163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6448.569323] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6450.198765] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6451.276762] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6453.301654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6453.418366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6455.098991] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6456.681568] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6460.969780] loop0: detected capacity change from 0 to 204800000 [ 6461.080105] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6461.082934] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6461.086049] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6461.109459] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6462.888089] loop0: detected capacity change from 0 to 204800000 [ 6462.937476] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6462.940340] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6462.943271] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6462.991615] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6464.976618] loop0: detected capacity change from 0 to 204800000 [ 6467.239476] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6467.270698] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6467.334899] Lustre: fs15246-OST0000: new disk, initializing [ 6467.337573] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 6467.363619] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6469.493974] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6473.798075] loop1: detected capacity change from 0 to 204800000 [ 6475.962316] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6475.994735] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6476.032345] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 6476.063293] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 6476.098493] Lustre: fs15246-MDT0000: new disk, initializing [ 6476.183949] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6476.198410] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6476.259662] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6477.281390] Lustre: 511297:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000489b003a x1849564839571584/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763882539 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 6477.723100] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6486.178168] LustreError: 511297:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 6486.184671] LustreError: 511297:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 6486.190271] LustreError: 511297:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 6490.089459] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6490.093908] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 6492.281846] LustreError: 512422:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 6492.306419] Lustre: server umount fs15246-MDT0000 complete [ 6495.207984] LustreError: 508398:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6495.211350] LustreError: 508398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6495.771167] LustreError: 512923:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 6495.775159] LustreError: 512923:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6495.795970] Lustre: server umount fs15246-OST0000 complete [ 6500.326645] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6500.328775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6500.333840] Lustre: Skipped 1 previous similar message [ 6505.445964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6505.449489] Lustre: Skipped 2 previous similar messages [ 6505.648213] LustreError: 513423:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6505.650448] LustreError: 513423:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6505.704855] Lustre: server umount lustre-OST0000 complete [ 6509.024587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6509.026586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6509.030032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6513.291557] LustreError: 513625:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6513.293423] LustreError: 513625:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6513.332508] Lustre: server umount lustre-MDT0000 complete [ 6514.789609] LustreError: 506951:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763882560 with bad export cookie 4000885313251556028 [ 6514.795763] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6514.951297] Lustre: server umount lustre-MDT0001 complete [ 6518.668502] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 6519.664606] Key type lgssc unregistered [ 6519.802390] LNet: 514359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6519.804678] LNetError: 514359:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6519.810482] LNet: Removed LNI 192.168.204.127@tcp [ 6520.105534] Key type .llcrypt unregistered [ 6520.106483] Key type ._llcrypt unregistered [ 6528.033689] Key type ._llcrypt registered [ 6528.034782] Key type .llcrypt registered [ 6528.079417] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 6528.465966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6528.527941] alg: No test for adler32 (adler32-zlib) [ 6529.428294] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 6529.531791] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 6531.120155] Key type lgssc registered [ 6531.647611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6534.618717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6536.548740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6538.501591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6540.382259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6546.413897] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 6550.079801] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 6553.996558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6556.708752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6559.379296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6561.972961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6565.008596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6565.038855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6565.048333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6566.171490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6566.189753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6566.240566] Lustre: lustre-MDT0000: new disk, initializing [ 6566.270752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6566.278363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6567.893454] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6573.043300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6573.082183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6573.137188] Lustre: 518765:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6573.172423] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6573.175253] Lustre: Skipped 1 previous similar message [ 6573.217151] Lustre: lustre-MDT0001: new disk, initializing [ 6573.274415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6573.290259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6573.294054] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6574.893057] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6578.523721] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6579.865285] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6581.907828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6581.928891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6582.012609] Lustre: lustre-OST0000: new disk, initializing [ 6582.014437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6582.036806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6582.771944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6582.775060] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6582.806329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6583.795329] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6586.900272] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6588.209072] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6588.269388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6589.465512] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6589.524690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6592.994889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6592.995732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6592.999158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6593.000754] LustreError: Skipped 1 previous similar message [ 6596.207067] LustreError: 521700:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6596.241399] Lustre: server umount lustre-OST0000 complete [ 6598.113685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6598.114379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6598.116632] Lustre: Skipped 2 previous similar messages [ 6598.118131] Lustre: Skipped 1 previous similar message [ 6603.232957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6603.234750] Lustre: Skipped 1 previous similar message [ 6603.515414] LustreError: 521901:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6603.516826] LustreError: 521901:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6603.540531] Lustre: server umount lustre-MDT0000 complete [ 6604.523978] LustreError: 518757:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763882650 with bad export cookie 8559260505148892018 [ 6604.526586] LustreError: 518757:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6604.527077] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6604.581567] LustreError: 522102:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6604.583041] LustreError: 522102:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6604.623577] Lustre: server umount lustre-MDT0001 complete [ 6611.015896] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 6613.688709] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 6616.343774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6618.231760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6619.914766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6621.810519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6622.333396] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6487 sec ========== 02:24:28 (1763882668) [ 6622.848823] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:24:28 (1763882668) === [ 6623.883550] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:24:29 (1763882669) === [ 6635.582875] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 6636.403860] Key type lgssc unregistered [ 6636.524339] LNet: 526947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6636.526626] LNetError: 526947:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6636.534293] LNet: Removed LNI 192.168.204.127@tcp [ 6636.784937] Key type .llcrypt unregistered [ 6636.786053] Key type ._llcrypt unregistered