[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 853464360 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001018] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002000] kvm-guest: setup PV IPIs [ 0.004449] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.005000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.005023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007009] pid_max: default: 32768 minimum: 301 [ 0.009124] LSM: Security Framework initializing [ 0.010060] Yama: becoming mindful. [ 0.011042] SELinux: Initializing. [ 0.011920] *** VALIDATE selinux *** [ 0.022177] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030337] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032097] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033362] *** VALIDATE tmpfs *** [ 0.035316] *** VALIDATE proc *** [ 0.036256] *** VALIDATE cgroup *** [ 0.037011] *** VALIDATE cgroup2 *** [ 0.038278] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039201] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041031] Spectre V2 : User space: Vulnerable [ 0.042009] Speculative Store Bypass: Vulnerable [ 0.045673] debug: unmapping init [mem 0xffffffff8d659000-0xffffffff8d660fff] [ 0.047158] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048677] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049023] ... version: 2 [ 0.050017] ... bit width: 48 [ 0.051008] ... generic registers: 4 [ 0.051948] ... value mask: 0000ffffffffffff [ 0.052023] ... max period: 00007fffffffffff [ 0.053022] ... fixed-purpose events: 3 [ 0.054020] ... event mask: 000000070000000f [ 0.055412] rcu: Hierarchical SRCU implementation. [ 0.059853] smp: Bringing up secondary CPUs ... [ 0.060776] x86: Booting SMP configuration: [ 0.061072] .... node #0, CPUs: #1 #2 #3 [ 0.076021] smp: Brought up 1 node, 4 CPUs [ 0.078021] smpboot: Max logical packages: 1 [ 0.079031] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139290] node 0 deferred pages initialised in 56ms [ 0.144036] devtmpfs: initialized [ 0.145000] x86/mm: Memory block size: 128MB [ 0.148722] gcov: version magic: 0x41383552 [ 0.152462] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156049] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160046] pinctrl core: initialized pinctrl subsystem [ 0.162294] [ 0.162913] ************************************************************* [ 0.165014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168016] ** ** [ 0.170016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171034] ** ** [ 0.172019] ** This means that this kernel is built to expose internal ** [ 0.174022] ** IOMMU data structures, which may compromise security on ** [ 0.175026] ** your system. ** [ 0.176016] ** ** [ 0.179016] ** If you see this message and you are not debugging the ** [ 0.181020] ** kernel, report this immediately to your vendor! ** [ 0.188021] ** ** [ 0.194020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198017] ************************************************************* [ 0.205542] NET: Registered protocol family 16 [ 0.208578] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.211104] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.213089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.217129] cpuidle: using governor menu [ 0.220220] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.225038] PCI: Using configuration type 1 for base access [ 0.227159] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.245113] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.246026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.252469] cryptd: max_cpu_qlen set to 1000 [ 0.259427] ACPI: Added _OSI(Module Device) [ 0.262022] ACPI: Added _OSI(Processor Device) [ 0.264024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.266022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.272505] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.281660] ACPI: Interpreter enabled [ 0.283103] ACPI: PM: (supports S0 S3 S4 S5) [ 0.284017] ACPI: Using IOAPIC for interrupt routing [ 0.287142] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.293642] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.307625] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.311083] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.316061] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.317125] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.324696] acpiphp: Slot [2] registered [ 0.326281] acpiphp: Slot [3] registered [ 0.328134] acpiphp: Slot [4] registered [ 0.329169] acpiphp: Slot [5] registered [ 0.331257] acpiphp: Slot [6] registered [ 0.333175] acpiphp: Slot [7] registered [ 0.334397] acpiphp: Slot [8] registered [ 0.337097] acpiphp: Slot [9] registered [ 0.338431] acpiphp: Slot [10] registered [ 0.341196] acpiphp: Slot [11] registered [ 0.343174] acpiphp: Slot [12] registered [ 0.344116] acpiphp: Slot [13] registered [ 0.350067] acpiphp: Slot [14] registered [ 0.351165] acpiphp: Slot [15] registered [ 0.354202] acpiphp: Slot [16] registered [ 0.355106] acpiphp: Slot [17] registered [ 0.357176] acpiphp: Slot [18] registered [ 0.359179] acpiphp: Slot [19] registered [ 0.361183] acpiphp: Slot [20] registered [ 0.363178] acpiphp: Slot [21] registered [ 0.365172] acpiphp: Slot [22] registered [ 0.367165] acpiphp: Slot [23] registered [ 0.369203] acpiphp: Slot [24] registered [ 0.370165] acpiphp: Slot [25] registered [ 0.373189] acpiphp: Slot [26] registered [ 0.374000] acpiphp: Slot [27] registered [ 0.374211] acpiphp: Slot [28] registered [ 0.376104] acpiphp: Slot [29] registered [ 0.378188] acpiphp: Slot [30] registered [ 0.380194] acpiphp: Slot [31] registered [ 0.381089] PCI host bridge to bus 0000:00 [ 0.383041] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.386045] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.389044] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.392035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.395037] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.397041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.400461] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.406000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.413722] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.429020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.432000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.433039] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.435032] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.439026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.443449] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.446786] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.449052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.453112] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.459020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.469024] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.476026] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.482341] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.490026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.498025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.515021] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.522963] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.530026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.542039] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.561023] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.580046] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.595027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.604031] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.625023] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.641189] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.652021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.665023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.705024] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.725790] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.741032] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.751028] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.783027] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.802042] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.816023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.828026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.858022] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.881000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.884000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.886441] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.887527] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.891312] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.894074] iommu: Default domain type: Passthrough [ 0.896943] SCSI subsystem initialized [ 0.898292] ACPI: bus type USB registered [ 0.902088] usbcore: registered new interface driver usbfs [ 0.904134] usbcore: registered new interface driver hub [ 0.907154] usbcore: registered new device driver usb [ 0.909698] pps_core: LinuxPPS API ver. 1 registered [ 0.912129] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.917080] PTP clock support registered [ 0.920369] EDAC MC: Ver: 3.0.0 [ 0.923152] PCI: Using ACPI for IRQ routing [ 0.926970] NetLabel: Initializing [ 0.928015] NetLabel: domain hash size = 128 [ 0.930014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.933352] NetLabel: unlabeled traffic allowed by default [ 0.937067] vgaarb: loaded [ 0.939595] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.942016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.951000] clocksource: Switched to clocksource kvm-clock [ 1.297653] VFS: Disk quotas dquot_6.6.0 [ 1.299327] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.301770] *** VALIDATE ramfs *** [ 1.313305] *** VALIDATE hugetlbfs *** [ 1.314777] pnp: PnP ACPI init [ 1.324228] pnp: PnP ACPI: found 6 devices [ 1.381091] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.393330] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.399949] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.406868] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.414345] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.417782] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.423619] NET: Registered protocol family 2 [ 1.426783] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.433157] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.438482] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.454525] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.466427] TCP: Hash tables configured (established 65536 bind 65536) [ 1.475273] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.485446] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.491425] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.498791] NET: Registered protocol family 1 [ 1.507692] RPC: Registered named UNIX socket transport module. [ 1.512550] RPC: Registered udp transport module. [ 1.513832] RPC: Registered tcp transport module. [ 1.520764] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.527573] NET: Registered protocol family 44 [ 1.531771] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.537069] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.540700] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.545768] PCI: CLS 0 bytes, default 64 [ 1.549919] Unpacking initramfs... [ 7.069623] debug: unmapping init [mem 0xffff930cbcc54000-0xffff930cbffbffff] [ 7.089742] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 7.103394] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 7.112786] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 9.527041] Initialise system trusted keyrings [ 9.536551] Key type blacklist registered [ 9.550870] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 9.561060] zbud: loaded [ 9.576099] *** VALIDATE nfs *** [ 9.577163] *** VALIDATE nfs4 *** [ 9.589977] pstore: using deflate compression [ 9.610049] Platform Keyring initialized [ 10.129941] NET: Registered protocol family 38 [ 10.131682] Key type asymmetric registered [ 10.133011] Asymmetric key parser 'x509' registered [ 10.142449] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 10.158258] io scheduler mq-deadline registered [ 10.159959] io scheduler kyber registered [ 10.161694] io scheduler bfq registered [ 10.179682] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 10.196446] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 10.233079] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 10.236156] ACPI: Power Button [PWRF] [ 10.665642] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 11.187397] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 12.022600] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 12.149506] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 12.682867] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 12.769954] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 12.818295] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 12.834440] Non-volatile memory driver v1.3 [ 12.836067] Linux agpgart interface v0.103 [ 12.990567] virtio_blk virtio1: [vda] 133160 512-byte logical blocks (68.2 MB/65.0 MiB) [ 13.001189] vda: detected capacity change from 0 to 68177920 [ 13.088778] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 13.107730] vdb: detected capacity change from 0 to 1073741824 [ 13.173487] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 13.192887] vdc: detected capacity change from 0 to 2621440000 [ 13.236231] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 13.254710] vdd: detected capacity change from 0 to 2621440000 [ 13.300345] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 13.311707] vde: detected capacity change from 0 to 4294967296 [ 13.352673] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 13.366662] vdf: detected capacity change from 0 to 4294967296 [ 13.417261] libphy: Fixed MDIO Bus: probed [ 13.441648] usbcore: registered new interface driver usbserial_generic [ 13.449273] usbserial: USB Serial support registered for generic [ 13.464409] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 13.480983] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 13.484603] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 13.494222] mousedev: PS/2 mouse device common for all mice [ 13.512548] rtc_cmos 00:05: RTC can wake from S4 [ 13.519957] rtc_cmos 00:05: registered as rtc0 [ 13.523569] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 13.543596] intel_pstate: CPU model not supported [ 13.571972] hid: raw HID events driver (C) Jiri Kosina [ 13.579629] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 13.586197] usbcore: registered new interface driver usbhid [ 13.611947] usbhid: USB HID core driver [ 13.618035] drop_monitor: Initializing network drop monitor service [ 13.620406] Initializing XFRM netlink socket [ 13.623617] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 13.645345] NET: Registered protocol family 10 [ 13.691683] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 13.702483] Segment Routing with IPv6 [ 13.708712] NET: Registered protocol family 17 [ 13.710939] mpls_gso: MPLS GSO support [ 13.715923] RAS: Correctable Errors collector initialized. [ 13.717973] AVX version of gcm_enc/dec engaged. [ 13.719738] AES CTR mode by8 optimization enabled [ 13.930260] sched_clock: Marking stable (13930242091, 0)->(16460204779, -2529962688) [ 13.945721] registered taskstats version 1 [ 13.951869] Loading compiled-in X.509 certificates [ 13.960527] zswap: loaded using pool lzo/zbud [ 14.045271] Key type big_key registered [ 14.080307] Key type encrypted registered [ 14.082427] ima: No TPM chip found, activating TPM-bypass! [ 14.084146] ima: Allocated hash algorithm: sha1 [ 14.085700] ima: No architecture policies found [ 14.091392] evm: Initialising EVM extended attributes: [ 14.093738] evm: security.selinux [ 14.096191] evm: security.ima [ 14.098951] evm: security.capability [ 14.100272] evm: HMAC attrs: 0x1 [ 14.107140] rtc_cmos 00:05: setting system clock to 2025-09-24 18:03:26 UTC (1758737006) [ 14.115216] debug: unmapping init [mem 0xffffffff8e603000-0xffffffff8e7fffff] [ 14.118953] debug: unmapping init [mem 0xffffffff8d382000-0xffffffff8d658fff] [ 14.130248] Write protecting the kernel read-only data: 28672k [ 14.136764] debug: unmapping init [mem 0xffffffff8ba03000-0xffffffff8bbfffff] [ 14.139563] debug: unmapping init [mem 0xffffffff8c314000-0xffffffff8c3fffff] [ 14.275428] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.298357] systemd[1]: Detected virtualization kvm. [ 14.304951] systemd[1]: Detected architecture x86-64. [ 14.308802] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 14.369467] systemd[1]: No hostname configured. [ 14.389592] systemd[1]: Set hostname to . [ 14.395858] random: systemd: uninitialized urandom read (16 bytes read) [ 14.410545] systemd[1]: Initializing machine ID from random generator. [ 14.537202] random: ln: uninitialized urandom read (6 bytes read) [ 14.798078] random: systemd: uninitialized urandom read (16 bytes read) [ 14.807608] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 14.830408] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 14.848251] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 16.967392] device-mapper: uevent: version 1.0.3 [ 16.969592] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 19.010296] virtio_net virtio0 ens2: renamed from eth0 [ 19.014446] random: fast init done [ 19.591788] scsi host0: ata_piix [ 19.650763] scsi host1: ata_piix [ 19.652523] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 19.661961] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 24.721600] random: crng init done [ 24.722983] random: 7 urandom warning(s) missed due to ratelimiting [ 27.866507] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 29.579141] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 32.685713] printk: systemd: 26 output lines suppressed due to ratelimiting [ 33.746862] SELinux: Disabled at runtime. [ 33.991996] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 34.048708] systemd[1]: Detected virtualization kvm. [ 34.057372] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 36.660923] systemd[1]: initrd-switch-root.service: Succeeded. [ 36.676875] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 36.696118] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 36.715074] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 36.730597] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 36.758286] systemd[1]: Starting Journal Service... Starting Journal Service... [ 36.834868] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ 37.014645] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 38.691075] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 40.107536] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 40.199085] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 42.246465] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 42.394692] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit)[ 48.036269] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 48.980494] NFS: Registering the id_resolver key type [ 48.982557] Key type id_resolver registered [ 48.984159] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ 50.264245] hrtimer: interrupt took 2356011 ns [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Login Service. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg440-server login: [ 97.442336] libcfs: loading out-of-tree module taints kernel. [ 97.457144] Key type ._llcrypt registered [ 97.458642] Key type .llcrypt registered [ 97.515926] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 110.306324] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 111.164757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 111.179982] alg: No test for adler32 (adler32-zlib) [ 112.367824] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 112.834467] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 114.511257] Key type lgssc registered [ 115.404823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 127.226273] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 129.322327] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.184900] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 141.747414] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 146.748897] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 157.053511] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 166.523300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 166.568803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 166.588440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.761378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 167.785140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 167.850286] Lustre: lustre-MDT0000: new disk, initializing [ 167.942645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 167.960093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 170.703468] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.032025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 180.084340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 180.138687] Lustre: 6510:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 180.168680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 180.171438] Lustre: Skipped 1 previous similar message [ 180.228606] Lustre: lustre-MDT0001: new disk, initializing [ 180.277560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 180.297065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 180.302885] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 182.781593] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.973465] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 192.429870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 192.475840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 192.605968] Lustre: lustre-OST0000: new disk, initializing [ 192.609074] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 192.666653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.885520] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.197466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 200.208432] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 200.296324] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 205.826424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.900063] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 205.989885] Lustre: lustre-OST0001: new disk, initializing [ 206.004325] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 206.098812] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 206.172798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 206.184432] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 206.240961] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 210.886533] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.577157] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.267042] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 235.240233] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing check_logdir /tmp/testlogs/ [ 239.250814] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing yml_node [ 243.909344] Lustre: DEBUG MARKER: Client: 2.16.58.53 [ 245.975754] Lustre: DEBUG MARKER: MDS: 2.16.58.53 [ 248.097962] Lustre: DEBUG MARKER: OSS: 2.16.58.53 [ 249.259040] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 24 14:07:20 EDT 2025 [ 263.350668] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 264.732528] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 282.592164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 282.598996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 282.607658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 283.105992] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 283.108302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 283.116660] Lustre: Skipped 2 previous similar messages [ 283.123687] Lustre: Skipped 2 previous similar messages [ 284.245120] LustreError: 12213:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 284.293515] Lustre: server umount lustre-MDT0000 complete [ 287.028602] LustreError: 6502:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737279 with bad export cookie 591538162292143142 [ 287.032217] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 287.046803] LustreError: 6502:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 288.223940] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.224610] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 288.229947] Lustre: Skipped 1 previous similar message [ 288.238376] Lustre: Skipped 2 previous similar messages [ 292.257352] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 292.260281] Lustre: Skipped 1 previous similar message [ 293.239198] LustreError: 12414:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 293.244363] LustreError: 12414:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 293.394288] Lustre: server umount lustre-MDT0001 complete [ 296.318523] LustreError: 12615:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 296.328251] LustreError: 12615:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 296.370319] Lustre: server umount lustre-OST0000 complete [ 298.952515] LustreError: 12816:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 298.955429] LustreError: 12816:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 299.120650] Lustre: server umount lustre-OST0001 complete [ 303.704311] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 311.224836] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 321.737346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 330.506943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 339.021367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 344.970915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 351.621407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 351.723635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 351.920751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 351.966916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 352.062762] Lustre: lustre-MDT0000: new disk, initializing [ 352.186394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 352.220283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 356.919996] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 366.445726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 366.525724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 366.605360] Lustre: 15003:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 366.659467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 366.662494] Lustre: Skipped 1 previous similar message [ 366.708650] Lustre: lustre-MDT0001: new disk, initializing [ 366.823131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 366.865550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 366.881882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 370.257440] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 376.771312] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 379.766984] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 385.359619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 385.445388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 385.644744] Lustre: lustre-OST0000: new disk, initializing [ 385.653930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 385.727473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 387.507269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 387.516076] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 387.643883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 390.576122] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 397.265183] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 401.027217] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 401.247540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 404.541863] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 404.714909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 408.035285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 408.037750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 408.052910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 412.532224] LustreError: 17942:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 412.538064] LustreError: 17942:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 412.663744] Lustre: server umount lustre-OST0000 complete [ 418.272907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 418.274168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 418.285174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 418.292827] Lustre: Skipped 2 previous similar messages [ 418.295574] Lustre: Skipped 1 previous similar message [ 421.295904] LustreError: 18145:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 421.325645] LustreError: 18145:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 421.425923] Lustre: server umount lustre-MDT0000 complete [ 423.398781] LustreError: 15867:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 423.412419] LustreError: 15867:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 6 previous similar messages [ 424.034123] LustreError: 14995:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737416 with bad export cookie 591538162292144675 [ 424.038066] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 424.045783] LustreError: 14995:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 424.351163] Lustre: server umount lustre-MDT0001 complete [ 427.613853] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 14:10:18 (1758737418) [ 429.474725] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 430.769585] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 14:10:21 (1758737421) [ 435.379706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.677167] LustreError: 19017:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 435.740810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.075540] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 440.804765] LustreError: 19017:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.127504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 446.970700] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 450.523792] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 452.661474] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 457.096443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 461.075787] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 464.241643] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 475.111678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 475.113308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.120577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 475.127206] Lustre: Skipped 1 previous similar message [ 475.137025] Lustre: Skipped 1 previous similar message [ 478.206221] LustreError: 21259:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 478.216151] LustreError: 21259:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 478.370023] Lustre: server umount lustre-OST0000 complete [ 485.344321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 487.367363] Lustre: server umount lustre-MDT0000 complete [ 490.466545] LustreError: 20224:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.490507] LustreError: 20224:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 490.589432] LustreError: 18998:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737482 with bad export cookie 591538162292145557 [ 490.598534] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 490.609324] LustreError: 18998:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 490.932599] Lustre: server umount lustre-MDT0001 complete [ 499.708888] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 502.533859] Key type lgssc unregistered [ 502.867653] LNet: 22196:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 502.880273] LNetError: 22196:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 502.900024] LNet: Removed LNI 192.168.204.140@tcp [ 503.811157] Key type .llcrypt unregistered [ 503.814794] Key type ._llcrypt unregistered [ 521.698576] Key type ._llcrypt registered [ 521.700351] Key type .llcrypt registered [ 521.879952] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 522.727383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 522.853590] alg: No test for adler32 (adler32-zlib) [ 523.899143] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 524.124955] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 525.775282] Key type lgssc registered [ 526.693214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 532.878399] loop: module loaded [ 532.993932] loop0: detected capacity change from 0 to 204800000 [ 533.179784] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 533.187106] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 533.417633] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 533.438613] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 533.490489] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 533.537296] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 533.566340] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 533.618060] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 538.392017] loop0: detected capacity change from 0 to 204800000 [ 538.528725] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 538.535835] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 538.546167] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 538.641972] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 542.436866] loop0: detected capacity change from 0 to 204800000 [ 546.704701] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 546.779959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 546.796711] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.000578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 548.051387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 548.191526] Lustre: lustre-MDT0000: new disk, initializing [ 548.295163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 548.326622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 551.361507] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 558.896070] loop1: detected capacity change from 0 to 204800000 [ 564.481628] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 564.529161] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 564.759308] Lustre: lustre-OST07c6: new disk, initializing [ 564.764147] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 564.766847] Lustre: Skipped 1 previous similar message [ 564.838240] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 569.510786] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 571.984964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 571.999296] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 572.038772] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 658.912195] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 658.942095] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 664.036180] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 664.320246] LustreError: 26522:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 664.448625] Lustre: server umount lustre-OST07c6 complete [ 671.604578] LustreError: 27024:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 671.615357] LustreError: 27024:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 671.929118] Lustre: server umount lustre-MDT0000 complete [ 685.515806] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 687.982143] Key type lgssc unregistered [ 688.223940] LNet: 27855:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.230665] LNetError: 27855:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 688.246535] LNet: Removed LNI 192.168.204.140@tcp [ 688.974809] Key type .llcrypt unregistered [ 688.976287] Key type ._llcrypt unregistered [ 704.165781] Key type ._llcrypt registered [ 704.167509] Key type .llcrypt registered [ 704.275641] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 704.953393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 705.042734] alg: No test for adler32 (adler32-zlib) [ 706.082800] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 706.326376] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 707.967216] Key type lgssc registered [ 708.950442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 714.444967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 714.463228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.806993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 718.523562] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 725.186579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 725.473553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 728.101417] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 731.398431] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 733.894755] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 738.814075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 739.116289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 741.234282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 744.491420] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 748.805475] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 754.167114] LustreError: 31286:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 761.824387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 761.828643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 761.837794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 765.921765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 765.931880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 765.941318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 767.230252] LustreError: 31536:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 767.329403] Lustre: server umount lustre-OST0000 complete [ 771.041908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 771.045973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 771.052090] Lustre: Skipped 1 previous similar message [ 771.069289] Lustre: Skipped 2 previous similar messages [ 776.162614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 776.167370] Lustre: Skipped 1 previous similar message [ 776.280849] LustreError: 31738:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 776.286641] LustreError: 31738:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 776.334981] Lustre: server umount lustre-MDT0000 complete [ 778.652103] LustreError: 29073:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737771 with bad export cookie 6888274197135834293 [ 778.656070] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 778.676299] LustreError: 29073:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 778.790674] LustreError: 31940:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 778.796455] LustreError: 31940:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 779.001693] Lustre: server umount lustre-MDT0001 complete [ 787.115586] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 789.751611] Key type lgssc unregistered [ 790.004966] LNet: 32471:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.012670] LNetError: 32471:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.024234] LNet: Removed LNI 192.168.204.140@tcp [ 790.774655] Key type .llcrypt unregistered [ 790.776900] Key type ._llcrypt unregistered [ 807.106638] Key type ._llcrypt registered [ 807.108771] Key type .llcrypt registered [ 807.211583] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 808.070538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 808.090126] alg: No test for adler32 (adler32-zlib) [ 809.174097] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 809.470780] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 811.231189] Key type lgssc registered [ 812.312278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 819.445626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 819.466071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 821.141039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 825.619770] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 831.460054] LustreError: 33742:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 831.467292] LustreError: 33742:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 833.525914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 833.846706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 837.317269] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 841.738281] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 844.766522] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 850.656388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 851.040892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 855.998240] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 860.036192] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 866.424140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 874.978157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 874.987143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 875.017340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 877.025925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.026093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 877.037807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 877.038709] Lustre: Skipped 1 previous similar message [ 879.606068] LustreError: 36010:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 879.771738] Lustre: server umount lustre-OST0000 complete [ 885.217488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 885.221137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 885.234573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 887.266625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.268107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 887.286592] Lustre: Skipped 1 previous similar message [ 889.082756] LustreError: 36211:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 889.084716] LustreError: 36211:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 889.183497] Lustre: server umount lustre-MDT0000 complete [ 892.339376] LustreError: 33724:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737884 with bad export cookie 9758557959981995682 [ 892.348624] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 892.481589] LustreError: 36412:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 892.489369] LustreError: 36412:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 892.717412] Lustre: server umount lustre-MDT0001 complete [ 902.223138] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 904.648435] Key type lgssc unregistered [ 904.902107] LNet: 36947:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.917292] LNetError: 36947:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.931986] LNet: Removed LNI 192.168.204.140@tcp [ 905.704089] Key type .llcrypt unregistered [ 905.711960] Key type ._llcrypt unregistered [ 926.850416] Key type ._llcrypt registered [ 926.854076] Key type .llcrypt registered [ 926.949550] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 928.225480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 928.245691] alg: No test for adler32 (adler32-zlib) [ 929.263912] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 929.514149] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 931.183177] Key type lgssc registered [ 932.271594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 938.829126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 938.860189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 940.294937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 943.661272] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 949.730915] LustreError: 38344:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 949.750322] LustreError: 38344:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 950.238729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 950.626093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 953.954698] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 958.743535] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 962.150773] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 968.635897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 968.914601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 973.202942] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 977.607385] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 985.333047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 993.393677] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 994.634162] LustreError: 40611:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 994.751168] Lustre: server umount lustre-MDT0000 complete [ 995.820866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 995.823221] LustreError: 38341:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 995.827320] Lustre: Skipped 1 previous similar message [ 998.025488] LustreError: 38325:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737990 with bad export cookie 10791034714728943176 [ 998.036155] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 998.039475] LustreError: 38325:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 1 previous similar message [ 998.491856] LustreError: 39721:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.507271] LustreError: 39721:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 998.511579] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.40@tcp (stopping) [ 1000.930258] LustreError: 39721:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1000.931237] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.954314] Lustre: Skipped 1 previous similar message [ 1000.957914] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1003.622177] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.40@tcp (stopping) [ 1004.219645] LustreError: 40814:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1004.222287] LustreError: 40814:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1004.454848] Lustre: server umount lustre-MDT0001 complete [ 1024.253428] LustreError: 41038:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1024.261673] LustreError: 41038:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1024.542354] Lustre: server umount lustre-OST0000 complete [ 1038.502029] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1040.838579] Key type lgssc unregistered [ 1041.092629] LNet: 41876:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.105461] LNetError: 41876:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.123512] LNet: Removed LNI 192.168.204.140@tcp [ 1041.667731] Key type .llcrypt unregistered [ 1041.670369] Key type ._llcrypt unregistered [ 1060.537230] Key type ._llcrypt registered [ 1060.539192] Key type .llcrypt registered [ 1060.612682] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1061.308866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1061.320413] alg: No test for adler32 (adler32-zlib) [ 1062.347203] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 1062.524932] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1064.184131] Key type lgssc registered [ 1065.191488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1072.746807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1072.761164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1074.234661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1078.637236] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1084.388228] LustreError: 43253:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1084.406460] LustreError: 43253:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1087.511526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1087.999150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1091.217105] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1095.203111] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1097.923504] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1103.339976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1103.612107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.608766] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1113.253071] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1118.960698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1124.322141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1124.324172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.329935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1124.338979] Lustre: Skipped 1 previous similar message [ 1124.984584] Lustre: lustre-OST0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 1124.988448] Lustre: Skipped 1 previous similar message [ 1129.449234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1129.463045] Lustre: Skipped 1 previous similar message [ 1129.670934] LustreError: 45496:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1129.792281] Lustre: server umount lustre-OST0000 complete [ 1149.408212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1149.411844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.433651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1151.695745] LustreError: 45874:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1151.698555] LustreError: 45874:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1151.756121] Lustre: server umount lustre-MDT0000 complete [ 1154.017589] LustreError: 44867:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1154.024985] LustreError: 44867:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 1154.171040] LustreError: 43234:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738146 with bad export cookie 9745903657192590828 [ 1154.175607] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1154.177292] LustreError: 43234:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1160.336345] LustreError: 46076:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1160.339601] LustreError: 46076:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1160.467412] Lustre: server umount lustre-MDT0001 complete [ 1167.300971] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1169.682918] Key type lgssc unregistered [ 1170.049727] LNet: 46608:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1170.060649] LNetError: 46608:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1170.073608] LNet: Removed LNI 192.168.204.140@tcp [ 1170.779614] Key type .llcrypt unregistered [ 1170.781961] Key type ._llcrypt unregistered [ 1189.261416] Key type ._llcrypt registered [ 1189.263120] Key type .llcrypt registered [ 1189.356780] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1189.980340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1190.001772] alg: No test for adler32 (adler32-zlib) [ 1190.998607] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 1191.179726] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1192.807267] Key type lgssc registered [ 1193.987613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1200.041696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1200.065218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1201.543662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1205.502899] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1210.853454] LustreError: 47984:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1210.864076] LustreError: 47984:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1215.721977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1215.977142] LustreError: 47984:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1216.362941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1219.724898] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1225.172476] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1228.335065] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1234.329794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1234.712729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1240.081863] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1246.008102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1246.173446] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1251.158754] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1254.555820] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1266.493301] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-09-2414h24m16s [ 1267.711936] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1271.779614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1271.782043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1271.785778] Lustre: Skipped 1 previous similar message [ 1271.800799] Lustre: Skipped 1 previous similar message [ 1272.870573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1272.873466] Lustre: Skipped 1 previous similar message [ 1274.521936] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 1274.530839] Lustre: Skipped 2 previous similar messages [ 1275.624284] LustreError: 50549:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1275.695932] Lustre: server umount lustre-MDT0000 complete [ 1278.930933] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1278.951620] LustreError: 48697:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.957994] LustreError: 48697:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1283.813563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1283.885531] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1284.078453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1284.137219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1285.094969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1287.333421] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1289.930964] Lustre: DEBUG MARKER: Wait for df (36019) ... [ 1291.254583] Lustre: DEBUG MARKER: done [ 1294.306446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1294.318708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.337786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1295.336074] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.461017] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1300.470738] Lustre: Skipped 2 previous similar messages [ 1308.129664] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1308.169024] LustreError: 51604:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1308.171404] LustreError: 51604:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1308.358609] Lustre: server umount lustre-OST0000 complete [ 1315.811111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1315.815101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1315.830781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1315.834325] Lustre: Skipped 3 previous similar messages [ 1317.601178] LustreError: 51807:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1317.603175] LustreError: 51807:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1317.695176] Lustre: server umount lustre-MDT0000 complete [ 1320.764619] LustreError: 48705:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738313 with bad export cookie 10369649324972004912 [ 1320.766314] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1320.810471] LustreError: 48705:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1320.905074] LustreError: 52009:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1320.908124] LustreError: 52009:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1321.106507] Lustre: server umount lustre-MDT0001 complete [ 1328.608686] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1331.003763] Key type lgssc unregistered [ 1331.259132] LNet: 52541:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1331.273233] LNetError: 52541:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1331.289716] LNet: Removed LNI 192.168.204.140@tcp [ 1332.058275] Key type .llcrypt unregistered [ 1332.063864] Key type ._llcrypt unregistered [ 1359.373119] Key type ._llcrypt registered [ 1359.374513] Key type .llcrypt registered [ 1359.472558] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 1371.204436] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1372.050385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1372.072689] alg: No test for adler32 (adler32-zlib) [ 1373.023273] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 1373.199784] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1374.887148] Key type lgssc registered [ 1375.628844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1382.172980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1386.598133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1390.664443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1395.266027] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1400.405868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1400.458463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1400.474485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1401.699632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1401.724598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1401.771289] Lustre: lustre-MDT0000: new disk, initializing [ 1401.848677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1401.865755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1404.782920] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1412.504405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1412.576088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1412.664536] Lustre: 56469:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1412.740716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1412.743873] Lustre: Skipped 1 previous similar message [ 1412.823455] Lustre: lustre-MDT0001: new disk, initializing [ 1412.930439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1412.975304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1412.982183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1416.160440] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1422.260111] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1424.927272] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1430.228308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1430.287781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1430.461994] Lustre: lustre-OST0000: new disk, initializing [ 1430.468492] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1430.548713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1435.387613] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1437.773419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1437.805697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1437.959723] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1441.925908] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1446.391500] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1446.592491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1450.265401] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1450.534947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1453.545795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1453.549114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.557962] LustreError: Skipped 1 previous similar message [ 1453.578062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1453.588148] Lustre: Skipped 1 previous similar message [ 1458.360961] LustreError: 59409:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1458.545349] Lustre: server umount lustre-OST0000 complete [ 1463.776416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1463.803446] Lustre: Skipped 1 previous similar message [ 1463.817829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1463.830213] Lustre: Skipped 1 previous similar message [ 1468.807940] LustreError: 59610:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1468.817243] LustreError: 59610:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1468.935635] Lustre: server umount lustre-MDT0000 complete [ 1472.309104] LustreError: 58224:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738464 with bad export cookie 2428007715221732451 [ 1472.310307] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1472.317603] LustreError: 58224:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1472.421556] LustreError: 59813:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1472.425484] LustreError: 59813:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1472.681603] Lustre: server umount lustre-MDT0001 complete [ 1480.138987] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 14:27:51 (1758738471) [ 1486.069454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1486.529200] LustreError: 60484:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1486.552105] LustreError: 60484:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1486.618138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1490.463951] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1491.936820] LustreError: 60484:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1496.044675] LustreError: 60484:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.044577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1498.288763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1501.609451] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1506.569398] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1509.851291] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1515.662211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1515.924152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1520.372498] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1525.016893] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1540.598592] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-09-2414h28m51s [ 1541.667491] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1547.392319] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1549.323291] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1549.350890] LustreError: 60822:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1569.421917] Lustre: lustre-MDT0000: Client 1bd8bc3f-cc06-4cc2-a118-bc5278d43161 (at 192.168.204.40@tcp) reconnecting [ 1589.375106] LustreError: 60822:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1590.901185] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1592.873855] Lustre: DEBUG MARKER: done [ 1597.929948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1597.930407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1597.932077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1597.954774] Lustre: Skipped 2 previous similar messages [ 1602.766211] LustreError: 63507:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1602.775042] LustreError: 63507:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1602.898949] Lustre: server umount lustre-OST0000 complete [ 1606.113393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1606.121604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1606.148192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1606.150799] Lustre: Skipped 1 previous similar message [ 1612.315602] LustreError: 63709:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1612.318846] LustreError: 63709:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1612.411373] Lustre: server umount lustre-MDT0000 complete [ 1615.193337] LustreError: 63508:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738607 with bad export cookie 2428007715221733333 [ 1615.200853] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1615.202735] LustreError: 63508:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1615.547826] Lustre: server umount lustre-MDT0001 complete [ 1624.090082] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 1626.839363] Key type lgssc unregistered [ 1627.119705] LNet: 64444:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1627.136048] LNetError: 64444:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1627.158817] LNet: Removed LNI 192.168.204.140@tcp [ 1627.914791] Key type .llcrypt unregistered [ 1627.916443] Key type ._llcrypt unregistered [ 1657.018386] Key type ._llcrypt registered [ 1657.020341] Key type .llcrypt registered [ 1657.143361] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 1668.449182] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1669.237090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1669.352918] alg: No test for adler32 (adler32-zlib) [ 1670.376761] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 1670.569104] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 1672.215181] Key type lgssc registered [ 1673.082912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1679.688662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1684.503861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1688.534299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1692.320638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1697.363469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1697.429994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1697.461227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1698.648491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1698.671906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1698.733974] Lustre: lustre-MDT0000: new disk, initializing [ 1698.805439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1698.828814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1702.235755] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1710.649761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1710.722098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1710.790944] Lustre: 68372:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1710.874117] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1710.877859] Lustre: Skipped 1 previous similar message [ 1710.942707] Lustre: lustre-MDT0001: new disk, initializing [ 1711.021324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1711.068650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1711.082352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1714.457631] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1721.133495] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1724.291714] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1729.805282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1729.854609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1730.043589] Lustre: lustre-OST0000: new disk, initializing [ 1730.047567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1730.133459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1734.908641] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1740.394267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1740.414073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1740.490393] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1742.386151] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1746.552416] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1746.790784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1750.668784] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1750.947387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1756.130555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.137483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1756.140799] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1756.173874] Lustre: Skipped 1 previous similar message [ 1761.250541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1761.256666] Lustre: Skipped 1 previous similar message [ 1766.372982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1766.376019] Lustre: Skipped 1 previous similar message [ 1767.391746] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1767.433480] LustreError: 71310:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1767.685074] Lustre: server umount lustre-OST0000 complete [ 1772.514755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1772.519329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1772.527578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1772.534983] Lustre: Skipped 1 previous similar message [ 1776.609091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1776.616284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1776.642334] Lustre: Skipped 1 previous similar message [ 1777.761326] LustreError: 71513:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1777.772811] LustreError: 71513:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1777.834303] Lustre: server umount lustre-MDT0000 complete [ 1780.821713] LustreError: 70118:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738773 with bad export cookie 9766777074233499126 [ 1780.822694] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1780.835791] LustreError: 70118:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1780.977123] LustreError: 71714:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1780.984383] LustreError: 71714:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1781.323939] Lustre: server umount lustre-MDT0001 complete [ 1787.885823] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 14:32:58 (1758738778) [ 1788.975706] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1790.360988] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 14:33:01 (1758738781) [ 1799.581385] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 1803.332947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.846071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1804.785961] Lustre: Failing over lustre-MDT0000 [ 1804.791148] LustreError: 72792:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1804.797329] LustreError: 72792:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1804.802989] LustreError: 72792:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1804.916080] LustreError: 72843:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1804.921139] LustreError: 72843:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1805.142759] Lustre: server umount lustre-MDT0000 complete [ 1810.150951] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 14:33:21 (1758738801) [ 1814.536563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.828360] LustreError: 73411:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1814.851430] LustreError: 73411:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1814.883784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1817.340159] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1820.129260] LustreError: 73411:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1824.224383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1824.623785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1827.922435] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1833.139513] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1836.785361] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1843.257260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1843.608028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1847.887949] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1851.754524] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1866.302949] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1871.333721] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.334403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1871.364969] Lustre: Skipped 1 previous similar message [ 1871.370870] Lustre: Skipped 1 previous similar message [ 1873.079864] LustreError: 75774:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1873.095712] LustreError: 75774:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1873.188140] Lustre: server umount lustre-MDT0000 complete [ 1876.449060] LustreError: 73406:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1876.479552] LustreError: 73406:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1877.125130] LustreError: 73392:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738869 with bad export cookie 9766777074233500281 [ 1877.144708] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1877.676670] Lustre: server umount lustre-MDT0001 complete [ 1881.207376] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1886.031613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1893.747537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.927365] Lustre: 66584:0:(client.c:2464:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758738873/real 1758738873] req@0000000008cf3aa1 x1844171151319168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1758738889 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1896.984944] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1896.996584] Lustre: Skipped 1 previous similar message [ 1897.951260] Lustre: 66585:0:(client.c:2464:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758738874/real 1758738874] req@000000007dbd6358 x1844171151319296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1758738890 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1902.502275] LustreError: 75019:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1902.624193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.763713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1903.595615] Lustre: 66587:0:(client.c:2464:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758738879/real 1758738879] req@0000000091aeec46 x1844171151319424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1758738895 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1906.417785] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1906.674384] LustreError: 75019:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1907.684968] Lustre: 66587:0:(client.c:2464:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758738884/real 1758738884] req@00000000e295556e x1844171151319680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1758738900 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1912.737848] Lustre: 66585:0:(client.c:2464:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758738889/real 1758738889] req@000000003e42082a x1844171151319808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1758738905 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1913.723920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1917.084524] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1919.484538] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1921.394558] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1924.154564] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1926.631361] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1929.185547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1929.189084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.221643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1931.352769] LustreError: 78235:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1931.355817] LustreError: 78235:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1931.424465] Lustre: server umount lustre-MDT0000 complete [ 1931.745158] LustreError: 75019:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1931.757701] LustreError: 75019:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 6 previous similar messages [ 1934.898811] LustreError: 73392:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738927 with bad export cookie 9766777074233502185 [ 1934.912932] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1934.923357] LustreError: 73392:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1935.377553] Lustre: server umount lustre-MDT0001 complete [ 1941.400969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1949.737947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1953.184456] Lustre: 66586:0:(client.c:2464:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758738929/real 1758738929] req@0000000075971848 x1844171151342464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1758738945 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1953.195498] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.201203] Lustre: Skipped 2 previous similar messages [ 1960.359446] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x878a90009e1def45 [ 1960.380981] Lustre: MGC192.168.204.140@tcp: Connection restored to 0@lo (at 0@lo) [ 1960.616404] LustreError: 75019:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1960.754126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1963.762120] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1970.217436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1970.460442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1970.466576] Lustre: Skipped 2 previous similar messages [ 1972.524289] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1973.710165] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 1977.712912] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1980.433756] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1986.030528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1986.034307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1986.043948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1991.050715] LustreError: 80694:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1991.053684] LustreError: 80694:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1991.108095] Lustre: server umount lustre-MDT0000 complete [ 1993.196467] LustreError: 79293:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1993.220479] LustreError: 79293:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 6 previous similar messages [ 1993.722147] LustreError: 73390:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738986 with bad export cookie 9766777074233503557 [ 1993.727022] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1993.728710] LustreError: 73390:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1993.967763] Lustre: server umount lustre-MDT0001 complete [ 1999.091099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2002.153618] Lustre: DEBUG MARKER: files compared the same [ 2014.481445] Lustre: server umount lustre-OST0000 complete [ 2029.302470] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 2032.200835] Key type lgssc unregistered [ 2032.569630] LNet: 82326:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.581822] LNetError: 82326:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2032.606500] LNet: Removed LNI 192.168.204.140@tcp [ 2033.574925] Key type .llcrypt unregistered [ 2033.576616] Key type ._llcrypt unregistered [ 2054.421846] Key type ._llcrypt registered [ 2054.423142] Key type .llcrypt registered [ 2054.638262] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2055.875994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2055.891628] alg: No test for adler32 (adler32-zlib) [ 2057.022124] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 2057.212196] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 2058.887176] Key type lgssc registered [ 2059.950472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2066.714858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2066.736974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.153417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.662752] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug +malloc all [ 2077.674958] LustreError: 83704:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2077.691594] LustreError: 83704:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2079.809711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2080.057811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2083.178688] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug +malloc all [ 2088.545420] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2092.503907] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2099.251759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2099.488557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2104.976716] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug +malloc all [ 2110.323075] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2112.815147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2123.232869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.235845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2123.264022] Lustre: Skipped 1 previous similar message [ 2123.283713] Lustre: Skipped 1 previous similar message [ 2127.292432] LustreError: 85950:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2127.478747] Lustre: server umount lustre-OST0000 complete [ 2131.423737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2131.426366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.439132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2133.474816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2133.478790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2133.498468] Lustre: Skipped 1 previous similar message [ 2136.835558] LustreError: 86152:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2136.843091] LustreError: 86152:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2137.001218] Lustre: server umount lustre-MDT0000 complete [ 2138.591755] LustreError: 83704:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2138.602274] LustreError: 83704:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2140.052918] LustreError: 83685:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758739132 with bad export cookie 7704882309867875223 [ 2140.056150] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2140.080071] LustreError: 83685:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2140.196669] LustreError: 86353:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2140.205926] LustreError: 86353:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2140.399437] Lustre: server umount lustre-MDT0001 complete [ 2149.192748] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 2151.685588] Key type lgssc unregistered [ 2152.064287] LNet: 86888:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.075586] LNetError: 86888:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2153.128278] LNet: Removed LNI 192.168.204.140@tcp [ 2153.933376] Key type .llcrypt unregistered [ 2153.935054] Key type ._llcrypt unregistered [ 2171.955411] Key type ._llcrypt registered [ 2171.957047] Key type .llcrypt registered [ 2172.045391] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2173.168051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2173.184238] alg: No test for adler32 (adler32-zlib) [ 2174.211391] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 2174.450853] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 2176.149180] Key type lgssc registered [ 2177.159884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2183.804149] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2183.822110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2201.440554] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2207.583743] LustreError: 88238:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.140@tcp: failed processing log, type 4: rc = -110 [ 2237.279393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2242.842215] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2248.389536] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2255.670392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2255.803296] Lustre: *** cfs_fail_loc=706, val=0*** [ 2256.258437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2256.334295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2259.722205] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2261.476609] LustreError: 88264:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2261.495578] LustreError: 88264:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2266.601298] LustreError: 88264:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2267.478934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2267.779380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2270.995572] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2275.838893] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2279.373339] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2281.963541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2281.968444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2281.982330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2282.481861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2282.482339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2287.367195] LustreError: 90563:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2287.574082] Lustre: server umount lustre-OST0000 complete [ 2292.705740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.708583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.717405] Lustre: Skipped 1 previous similar message [ 2292.727649] Lustre: Skipped 2 previous similar messages [ 2297.347759] LustreError: 90766:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2297.352969] LustreError: 90766:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2297.430947] Lustre: server umount lustre-MDT0000 complete [ 2297.830818] LustreError: 90565:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.844928] LustreError: 90565:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2300.157812] LustreError: 88246:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758739292 with bad export cookie 13509174831437777789 [ 2300.174352] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2300.269138] LustreError: 90967:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2300.273346] LustreError: 90967:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2300.459180] Lustre: server umount lustre-MDT0001 complete [ 2309.284438] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 2312.012727] Key type lgssc unregistered [ 2312.294806] LNet: 91499:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2312.300041] LNetError: 91499:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2312.385676] LNet: Removed LNI 192.168.204.140@tcp [ 2313.386594] Key type .llcrypt unregistered [ 2313.389017] Key type ._llcrypt unregistered [ 2334.522634] Key type ._llcrypt registered [ 2334.524368] Key type .llcrypt registered [ 2334.706549] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2335.906272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2336.017906] alg: No test for adler32 (adler32-zlib) [ 2337.138224] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 2337.395566] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 2339.111189] Key type lgssc registered [ 2340.273560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2347.094090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2347.112238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2351.584507] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2358.281147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2358.743326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2361.986405] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2363.874101] LustreError: 93545:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.892864] LustreError: 93545:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 2368.992904] LustreError: 93550:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.016368] LustreError: 93550:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2369.603175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2369.868480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2374.399947] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2375.140960] LustreError: 93551:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2375.161187] LustreError: 93551:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 2380.267448] LustreError: 93545:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.284361] LustreError: 93545:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2381.726488] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2381.748528] Lustre: MGS: Client 12b0d8be-e51e-4c3c-87a9-e1e46c13f4d9 (at 0@lo) reconnecting [ 2381.995637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2382.052113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2385.101955] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2397.665144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2397.675158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.694716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2399.713080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2399.713153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2399.741344] Lustre: Skipped 1 previous similar message [ 2403.012434] LustreError: 95507:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2403.233505] Lustre: server umount lustre-OST0000 complete [ 2409.955962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2409.956909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2409.959581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2409.988337] Lustre: Skipped 1 previous similar message [ 2415.072730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2415.076858] Lustre: Skipped 1 previous similar message [ 2420.197080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2421.215709] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2421.349633] LustreError: 95710:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2421.365463] LustreError: 95710:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2421.539103] Lustre: server umount lustre-MDT0000 complete [ 2425.402480] LustreError: 95915:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2425.412100] LustreError: 95915:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2425.686257] Lustre: server umount lustre-MDT0001 complete [ 2428.771361] LustreError: 96116:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2428.774198] LustreError: 96116:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2428.849172] Lustre: server umount MGS complete [ 2438.301889] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:43:49 (1758739429) [ 2467.435975] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 2476.256734] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2485.033057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2491.639051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2497.591268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2503.358132] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2509.526364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2509.595351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2513.371921] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2521.977323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2522.047654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2522.188501] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2522.224973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2522.301849] Lustre: lustre-MDT0001: new disk, initializing [ 2522.410193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2522.434726] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2522.449569] Lustre: 100634:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2522.464773] LustreError: 100634:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2526.217522] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2535.299510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2535.372535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2535.589781] Lustre: lustre-OST0000: new disk, initializing [ 2535.593588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2535.673584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2540.904359] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2542.113335] Lustre: 101732:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000006020c935 x1844171850684928/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1758739550 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2547.602448] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2552.777688] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2552.785393] Lustre: MGS: Client 20666514-7b88-4645-883c-207ae2bebffe (at 0@lo) reconnecting [ 2552.831277] Lustre: 99918:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2552.846343] LustreError: 99918:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2552.869789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2552.939713] Lustre: lustre-MDT0000: new disk, initializing [ 2553.049387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.079598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2553.181662] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2554.150402] Lustre: 102345:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@0000000030c7cbc9 x1844171850691328/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1758739562 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2556.255727] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2562.542689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2562.555313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2563.243129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2563.319906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2585.573547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2585.582551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2585.626279] Lustre: Skipped 1 previous similar message [ 2585.646160] Lustre: Skipped 2 previous similar messages [ 2599.391945] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2599.432948] LustreError: 102866:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2599.450048] LustreError: 102866:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2599.691975] Lustre: server umount lustre-OST0000 complete [ 2606.057305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2606.059829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2606.086326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2606.117348] Lustre: Skipped 4 previous similar messages [ 2609.268282] LustreError: 103069:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2609.279150] LustreError: 103069:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2609.289238] Lustre: server umount lustre-MDT0000 complete [ 2611.178443] LustreError: 102764:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.198785] LustreError: 102764:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2619.379533] Lustre: server umount lustre-MDT0001 complete [ 2622.638995] Lustre: server umount MGS complete [ 2634.000087] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:47:04 (1758739624) [ 2650.406980] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 2653.326430] Key type lgssc unregistered [ 2653.618896] LNet: 104813:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2653.628161] LNetError: 104813:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2653.647267] LNet: Removed LNI 192.168.204.140@tcp [ 2654.629629] Key type .llcrypt unregistered [ 2654.643037] Key type ._llcrypt unregistered [ 2670.827831] Key type ._llcrypt registered [ 2670.829599] Key type .llcrypt registered [ 2670.956087] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2671.660525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2671.799350] alg: No test for adler32 (adler32-zlib) [ 2672.865754] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 2673.046308] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 2674.719464] Key type lgssc registered [ 2675.965795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2683.079583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2683.107577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2684.733151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2689.288351] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2695.139841] LustreError: 106016:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.177800] LustreError: 106016:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2698.813036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2699.195637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2703.466690] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2709.845981] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2714.083253] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2721.531830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2721.868084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.784724] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2730.042757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2732.679952] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2740.205108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.209157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2740.226654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2740.272811] Lustre: Skipped 1 previous similar message [ 2745.315234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2745.325693] Lustre: Skipped 1 previous similar message [ 2745.594110] LustreError: 108261:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2745.801546] Lustre: server umount lustre-OST0000 complete [ 2750.436594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2750.440544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2750.441467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2750.474152] Lustre: Skipped 2 previous similar messages [ 2755.417622] LustreError: 108463:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2755.450133] LustreError: 108463:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2755.561937] LustreError: 107021:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.584260] LustreError: 107021:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2755.673989] Lustre: server umount lustre-MDT0000 complete [ 2759.367733] LustreError: 105997:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758739751 with bad export cookie 16397280562194366453 [ 2759.375904] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2759.384941] LustreError: 105997:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2759.583810] LustreError: 108665:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2759.595527] LustreError: 108665:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2759.870317] Lustre: server umount lustre-MDT0001 complete [ 2771.116142] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 2773.802121] Key type lgssc unregistered [ 2774.081956] LNet: 109198:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2774.088091] LNetError: 109198:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2774.115289] LNet: Removed LNI 192.168.204.140@tcp [ 2774.886770] Key type .llcrypt unregistered [ 2774.888427] Key type ._llcrypt unregistered [ 2788.078149] Key type ._llcrypt registered [ 2788.080384] Key type .llcrypt registered [ 2788.190828] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2789.254036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2789.307374] alg: No test for adler32 (adler32-zlib) [ 2790.431374] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 2790.736212] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 2792.455761] Key type lgssc registered [ 2793.670467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2798.748960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2798.756827] LustreError: 110239:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2798.758270] LustreError: 110242:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2798.766222] LustreError: 110239:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4999 [ 2798.770714] LustreError: 110239:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2798.780929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2798.783775] LustreError: 110239:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2800.185336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.726894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2807.031524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2810.226245] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2813.757920] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing lsmod [ 2815.798712] LustreError: 111411:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2815.859541] LustreError: 111422:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2815.868145] LustreError: 111411:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4931 [ 2815.871560] LustreError: 111411:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2815.891786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2815.897870] LustreError: 111411:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2816.267804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2822.628384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2822.639408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2822.644518] Lustre: Skipped 2 previous similar messages [ 2827.752778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2827.758343] Lustre: Skipped 2 previous similar messages [ 2832.870672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2832.874945] Lustre: Skipped 2 previous similar messages [ 2833.375739] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2833.475761] LustreError: 111606:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2833.556334] Lustre: server umount lustre-MDT0000 complete [ 2837.155048] LustreError: 110257:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758739829 with bad export cookie 5388121148849341941 [ 2837.160839] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2837.176828] LustreError: 110257:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2837.349265] LustreError: 111811:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2837.359214] LustreError: 111811:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2837.635818] Lustre: server umount lustre-MDT0001 complete [ 2851.205641] LustreError: 112012:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2851.212093] LustreError: 112012:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2851.498689] Lustre: server umount lustre-OST0000 complete [ 2858.590987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2858.972343] LustreError: 112490:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2858.978944] LustreError: 112490:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2859.049173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.962735] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2864.101714] LustreError: 112489:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2864.125071] LustreError: 112489:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2869.230605] LustreError: 112489:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.887702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2870.188915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2872.890635] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2877.194847] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2880.279269] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2885.804483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2886.100730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2891.195748] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2895.517634] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2904.559431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2914.793624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2914.794051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2914.803354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2914.818224] Lustre: Skipped 3 previous similar messages [ 2917.432154] LustreError: 114757:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2917.437987] LustreError: 114757:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2917.661191] Lustre: server umount lustre-OST0000 complete [ 2921.441948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2921.447509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.479814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2921.482621] Lustre: Skipped 1 previous similar message [ 2925.027347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2927.359258] LustreError: 114958:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2927.371486] LustreError: 114958:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2927.495105] Lustre: server umount lustre-MDT0000 complete [ 2930.145453] LustreError: 112490:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2930.172438] LustreError: 112490:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2930.761844] LustreError: 112471:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758739923 with bad export cookie 5388121148849342641 [ 2930.770165] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2931.216536] Lustre: server umount lustre-MDT0001 complete [ 2940.104176] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 2942.613489] Key type lgssc unregistered [ 2942.871743] LNet: 115692:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2942.875339] LNetError: 115692:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2942.896917] LNet: Removed LNI 192.168.204.140@tcp [ 2943.550951] Key type .llcrypt unregistered [ 2943.557783] Key type ._llcrypt unregistered [ 2962.631366] Key type ._llcrypt registered [ 2962.635765] Key type .llcrypt registered [ 2962.730715] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 2963.686951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2963.710675] alg: No test for adler32 (adler32-zlib) [ 2964.793929] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 2964.969730] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 2966.647959] Key type lgssc registered [ 2967.636813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2974.209995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2974.233612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.590415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2978.870229] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2984.929078] LustreError: 117067:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2984.944367] LustreError: 117067:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2986.887979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2987.192119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2990.510629] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 2995.338658] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2998.739933] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3004.476520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3004.871751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3010.046103] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3015.069389] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3020.332129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3025.050857] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3028.003069] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3032.548428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3032.552250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.591095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3032.953975] LustreError: 119416:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3033.198322] Lustre: server umount lustre-OST0000 complete [ 3038.689274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3038.704465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3038.715581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3040.736421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3040.736538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.856680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3045.865515] Lustre: Skipped 2 previous similar messages [ 3046.886211] LustreError: 119621:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3046.895821] LustreError: 119621:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3046.964226] Lustre: server umount lustre-MDT0000 complete [ 3050.004895] LustreError: 117047:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740042 with bad export cookie 2595453585929320566 [ 3050.006259] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3050.055595] LustreError: 117047:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3050.139705] LustreError: 119822:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3050.144639] LustreError: 119822:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3050.372737] Lustre: server umount lustre-MDT0001 complete [ 3059.445988] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 3062.109600] Key type lgssc unregistered [ 3062.410950] LNet: 120353:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3062.424220] LNetError: 120353:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3062.443219] LNet: Removed LNI 192.168.204.140@tcp [ 3063.192117] Key type .llcrypt unregistered [ 3063.198455] Key type ._llcrypt unregistered [ 3076.516120] Key type ._llcrypt registered [ 3076.517335] Key type .llcrypt registered [ 3076.609656] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3077.220753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3077.315337] alg: No test for adler32 (adler32-zlib) [ 3078.298549] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 3078.473401] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 3080.127207] Key type lgssc registered [ 3080.996894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3087.175154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3087.189317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3088.550261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3091.817943] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3098.085317] LustreError: 121538:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3098.099546] LustreError: 121538:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3098.858031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3099.108304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3101.889944] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3105.601665] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3108.702433] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3113.836273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3114.088831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3118.295481] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3121.924901] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3133.545382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3144.161200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3144.171147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3144.174324] Lustre: Skipped 1 previous similar message [ 3144.182723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3148.017315] LustreError: 123806:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3148.145583] Lustre: server umount lustre-OST0000 complete [ 3154.401492] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.415431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3154.419281] Lustre: Skipped 2 previous similar messages [ 3159.522079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3163.617178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3163.622256] Lustre: Skipped 2 previous similar messages [ 3165.151223] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3165.207555] LustreError: 124007:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3165.211690] LustreError: 124007:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3165.268324] Lustre: server umount lustre-MDT0000 complete [ 3167.664482] LustreError: 121518:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740160 with bad export cookie 11258475230246139107 [ 3167.666656] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3167.671262] LustreError: 121518:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3167.779540] LustreError: 124211:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3167.783397] LustreError: 124211:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3167.954287] Lustre: server umount lustre-MDT0001 complete [ 3174.318656] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 3176.256572] Key type lgssc unregistered [ 3176.478768] LNet: 124742:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3176.485439] LNetError: 124742:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3176.499883] LNet: Removed LNI 192.168.204.140@tcp [ 3177.064982] Key type .llcrypt unregistered [ 3177.067245] Key type ._llcrypt unregistered [ 3191.669188] Key type ._llcrypt registered [ 3191.671057] Key type .llcrypt registered [ 3191.783696] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3192.372090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3192.415649] alg: No test for adler32 (adler32-zlib) [ 3193.374529] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 3193.506990] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 3195.119208] Key type lgssc registered [ 3195.771189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3200.449218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3200.479748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3201.796181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3204.436591] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3210.044170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3210.324120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3212.640461] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3216.060891] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3218.431453] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3222.587595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3222.827680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3226.503129] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3226.949887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3230.289773] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3239.718122] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3242.380614] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3242.383699] Lustre: Skipped 2 previous similar messages [ 3245.398585] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3245.401603] Lustre: Skipped 2 previous similar messages [ 3247.634381] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3247.647583] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3257.583131] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.40@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3257.595586] Lustre: Skipped 2 previous similar messages [ 3266.587569] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.40@tcp 192.168.0.[2,10]@tcp [ 3270.020725] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.40/24@tcp in log lustre-MDT0000 [ 3270.024730] Lustre: Skipped 2 previous similar messages [ 3272.171557] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.40/24@tcp [ 3272.175448] Lustre: Skipped 1 previous similar message [ 3276.259265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3276.261821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3276.268243] Lustre: Skipped 1 previous similar message [ 3281.344849] LustreError: 129816:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3281.385127] LustreError: 127716:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3281.408581] LustreError: 127716:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3281.558666] Lustre: server umount lustre-OST0000 complete [ 3286.499587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3286.504117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3286.511734] Lustre: Skipped 1 previous similar message [ 3286.515857] Lustre: Skipped 1 previous similar message [ 3290.390846] LustreError: 130017:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3290.394403] LustreError: 130017:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3290.456436] Lustre: server umount lustre-MDT0000 complete [ 3291.617938] LustreError: 127326:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3291.633710] LustreError: 127326:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3293.262692] LustreError: 129116:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740285 with bad export cookie 15148019691395468829 [ 3293.265108] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3293.270055] LustreError: 129116:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3293.376590] LustreError: 130218:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3293.379609] LustreError: 130218:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3293.573327] Lustre: server umount lustre-MDT0001 complete [ 3300.461680] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 3302.725679] Key type lgssc unregistered [ 3302.953954] LNet: 130750:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3302.967032] LNetError: 130750:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3304.041411] LNet: Removed LNI 192.168.204.140@tcp [ 3304.662898] Key type .llcrypt unregistered [ 3304.664563] Key type ._llcrypt unregistered [ 3324.154189] Key type ._llcrypt registered [ 3324.155864] Key type .llcrypt registered [ 3324.237896] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3324.718359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3324.776425] alg: No test for adler32 (adler32-zlib) [ 3325.716571] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 3325.873377] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 3327.495692] Key type lgssc registered [ 3328.506956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3334.561900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3334.581900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3335.888173] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3335.912461] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.40/24@tcp [ 3335.926319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3338.596452] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3344.396290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3344.584591] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3344.604650] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.40/24@tcp [ 3344.647499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3347.085422] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3350.831576] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3353.415655] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3358.437909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3358.686635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3360.776569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3362.393368] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3365.899427] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3379.169022] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3379.173206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3379.177752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3379.180227] Lustre: Skipped 1 previous similar message [ 3379.185700] Lustre: Skipped 1 previous similar message [ 3382.969403] LustreError: 134572:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3383.109792] Lustre: server umount lustre-OST0000 complete [ 3389.408663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.409230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3389.414159] Lustre: Skipped 1 previous similar message [ 3394.528981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3394.531616] Lustre: Skipped 2 previous similar messages [ 3398.630463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3398.636628] Lustre: Skipped 1 previous similar message [ 3400.159691] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3400.224547] LustreError: 134774:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3400.227618] LustreError: 134774:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3400.276486] Lustre: server umount lustre-MDT0000 complete [ 3403.142271] LustreError: 132254:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740395 with bad export cookie 15576647541849752698 [ 3403.145943] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3403.149651] LustreError: 132254:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3403.273321] LustreError: 134977:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3403.285196] LustreError: 134977:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3403.461817] Lustre: server umount lustre-MDT0001 complete [ 3410.580480] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 3412.390472] Key type lgssc unregistered [ 3412.645183] LNet: 135509:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3412.651358] LNetError: 135509:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3412.661750] LNet: Removed LNI 192.168.204.140@tcp [ 3413.271514] Key type .llcrypt unregistered [ 3413.273120] Key type ._llcrypt unregistered [ 3429.621203] Key type ._llcrypt registered [ 3429.622316] Key type .llcrypt registered [ 3429.693754] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 3438.216920] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3438.712133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3438.780565] alg: No test for adler32 (adler32-zlib) [ 3439.763676] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 3439.912373] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 3441.538053] Key type lgssc registered [ 3442.405733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3448.320636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3451.846450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3455.131645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3458.401860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3462.668663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3462.714558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3462.738769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3463.926267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3463.947868] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3463.992854] Lustre: lustre-MDT0000: new disk, initializing [ 3464.044979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3464.058876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3466.134972] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3472.974897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3473.031844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3473.089138] Lustre: 138861:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3473.143522] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3473.146549] Lustre: Skipped 1 previous similar message [ 3473.195287] Lustre: lustre-MDT0001: new disk, initializing [ 3473.277833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3473.319256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3473.325481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3475.842405] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3480.818745] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3482.961540] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3486.877431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3486.919944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3487.083856] Lustre: lustre-OST0000: new disk, initializing [ 3487.087198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3487.152580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3488.346975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3488.351692] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3488.415256] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3490.875251] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3495.572326] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3497.912736] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3498.036577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3500.287616] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3500.416347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3504.097862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3504.098593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3504.102032] LustreError: Skipped 1 previous similar message [ 3504.114038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3504.119251] Lustre: Skipped 1 previous similar message [ 3509.219833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3514.340832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3514.348761] Lustre: Skipped 2 previous similar messages [ 3515.871154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3515.894826] LustreError: 141812:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3515.997037] Lustre: server umount lustre-OST0000 complete [ 3519.458279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3519.459711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3519.469364] Lustre: Skipped 2 previous similar messages [ 3519.476809] Lustre: Skipped 1 previous similar message [ 3524.231955] LustreError: 142015:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3524.234799] LustreError: 142015:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3524.283465] Lustre: server umount lustre-MDT0000 complete [ 3526.479564] LustreError: 138854:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740518 with bad export cookie 16227790094793588133 [ 3526.483593] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3526.484081] LustreError: 138854:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3526.591456] LustreError: 142218:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3526.593565] LustreError: 142218:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3526.726553] Lustre: server umount lustre-MDT0001 complete [ 3531.584778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3531.891866] LustreError: 142694:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3531.902314] LustreError: 142694:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3531.953935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3534.223259] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3537.377906] LustreError: 142694:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3539.048692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3539.281960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3541.504408] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3544.388747] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3546.464523] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3550.370261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3550.545638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3553.666697] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3556.794961] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3558.767744] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3558.891297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3561.018896] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3564.258409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 3569.869825] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3569.905199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3569.984902] Lustre: lustre-OST0001: new disk, initializing [ 3569.987741] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3570.039062] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3572.697056] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3576.351081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3576.357690] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3576.426141] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3577.054242] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3577.170095] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3579.006323] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3579.117459] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3580.992188] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8af4506e9000.ost_server_uuid 50 [ 3581.824850] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8af4506e9000.ost_server_uuid in FULL state after 0 sec [ 3586.531852] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3586.533016] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3586.539168] LustreError: Skipped 1 previous similar message [ 3586.554652] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3586.560514] Lustre: Skipped 1 previous similar message [ 3591.534220] LustreError: 146960:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3591.537765] LustreError: 146960:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3591.575908] Lustre: server umount lustre-OST0001 complete [ 3591.653435] LustreError: 144814:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3591.664528] LustreError: 144814:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3594.723170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3594.728861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3594.735087] Lustre: Skipped 1 previous similar message [ 3594.739736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3596.770781] LustreError: 144289:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3596.782325] LustreError: 144289:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3599.539204] LustreError: 147161:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3599.543089] LustreError: 147161:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3599.664320] Lustre: server umount lustre-OST0000 complete [ 3601.888634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3601.893057] Lustre: Skipped 2 previous similar messages [ 3607.648294] LustreError: 147362:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3607.654660] LustreError: 147362:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3607.697289] Lustre: server umount lustre-MDT0000 complete [ 3609.675454] LustreError: 142675:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740602 with bad export cookie 16227790094793589015 [ 3609.677832] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3609.680958] LustreError: 142675:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3609.894518] Lustre: server umount lustre-MDT0001 complete [ 3619.570662] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 3621.125178] Key type lgssc unregistered [ 3621.303700] LNet: 148553:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3621.307683] LNetError: 148553:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3621.318017] LNet: Removed LNI 192.168.204.140@tcp [ 3621.799350] Key type .llcrypt unregistered [ 3621.801205] Key type ._llcrypt unregistered [ 3639.137926] Key type ._llcrypt registered [ 3639.139624] Key type .llcrypt registered [ 3639.195646] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 3646.546311] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3646.941212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3646.995220] alg: No test for adler32 (adler32-zlib) [ 3647.878476] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 3647.977160] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 3649.575305] Key type lgssc registered [ 3650.140298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3654.701558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3658.042568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3660.918985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3663.920295] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3667.355441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3667.382251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3667.393572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3668.531215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3668.550469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3668.591896] Lustre: lustre-MDT0000: new disk, initializing [ 3668.643247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3668.659705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3670.796996] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3677.024778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3677.064218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3677.119261] Lustre: 152478:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3677.155561] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3677.157721] Lustre: Skipped 1 previous similar message [ 3677.197843] Lustre: lustre-MDT0001: new disk, initializing [ 3677.247496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3677.272146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3677.276627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3679.605643] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3684.076130] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3685.820241] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3689.317902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3689.352587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3689.486283] Lustre: lustre-OST0000: new disk, initializing [ 3689.488715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3689.528901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3692.328064] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3696.711282] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3697.698736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3697.703275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3697.756367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3698.717969] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3698.831661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3700.751936] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3700.865299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3702.755947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3702.756983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3702.758655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3702.760434] LustreError: Skipped 1 previous similar message [ 3702.766525] Lustre: Skipped 1 previous similar message [ 3703.866666] LustreError: 155416:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3703.962470] Lustre: server umount lustre-OST0000 complete [ 3707.872859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3707.874196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3707.879548] Lustre: Skipped 1 previous similar message [ 3707.883237] Lustre: Skipped 1 previous similar message [ 3707.884884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3712.992259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3712.995357] Lustre: Skipped 1 previous similar message [ 3718.113805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3718.116633] Lustre: Skipped 1 previous similar message [ 3720.159145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3720.208981] LustreError: 155617:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3720.211185] LustreError: 155617:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3720.250690] Lustre: server umount lustre-MDT0000 complete [ 3721.984027] LustreError: 152469:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740714 with bad export cookie 13217371085009360502 [ 3721.986861] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3721.989312] LustreError: 152469:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3722.066808] LustreError: 155821:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3722.069151] LustreError: 155821:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3722.170307] Lustre: server umount lustre-MDT0001 complete [ 3726.475950] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 15:05:17 (1758740717) [ 3734.470591] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 3738.516996] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3743.536626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3747.089121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3750.050513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3753.065661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3756.606258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3756.639292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3756.753288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3756.771743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3756.818949] Lustre: lustre-MDT0000: new disk, initializing [ 3756.866640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3756.879859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3758.677512] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3764.193052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3764.242156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3764.293733] Lustre: 158808:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3764.324844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3764.327666] Lustre: Skipped 1 previous similar message [ 3764.373584] Lustre: lustre-MDT0001: new disk, initializing [ 3764.418870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3764.438614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3764.443047] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3766.259558] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3770.182374] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3771.733504] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3775.065642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3775.098891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3775.236945] Lustre: lustre-OST0000: new disk, initializing [ 3775.239902] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3775.282115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3775.398294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3775.402718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3775.466698] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3777.942238] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3781.763048] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3784.047474] Lustre: Failing over lustre-OST0000 [ 3784.094079] LustreError: 161247:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3784.096164] LustreError: 161247:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3784.185255] Lustre: server umount lustre-OST0000 complete [ 3798.213972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3798.372420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3798.384791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3798.394788] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3798.627246] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3801.068896] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3802.447176] Lustre: 150693:0:(client.c:2464:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758740778/real 1758740778] req@000000009202a61f x1844173225387904/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1758740794 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3802.971111] Lustre: Failing over lustre-MDT0000 [ 3803.046096] LustreError: 162219:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3803.048404] LustreError: 162219:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3803.099047] Lustre: server umount lustre-MDT0000 complete [ 3803.620105] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.626112] Lustre: Skipped 2 previous similar messages [ 3806.687236] Lustre: 150694:0:(client.c:2464:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758740783/real 1758740783] req@000000000a14792d x1844173225388928/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1758740799 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3806.698716] Lustre: 150694:0:(client.c:2464:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3808.736802] LustreError: 158814:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3808.744720] LustreError: 158814:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 7 previous similar messages [ 3811.807281] Lustre: 150693:0:(client.c:2464:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758740788/real 1758740788] req@000000000dda907c x1844173225389952/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1758740804 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3813.856772] LustreError: 159603:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3813.868015] LustreError: 159603:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3817.122277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3817.196258] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3817.201789] Lustre: 158810:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.140@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3817.320711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3817.380140] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3819.095937] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3819.145937] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3822.565905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3822.575191] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3822.581365] Lustre: Skipped 1 previous similar message [ 3822.612867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3827.682424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3827.683595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3827.689460] LustreError: Skipped 1 previous similar message [ 3827.690983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3827.697074] Lustre: Skipped 2 previous similar messages [ 3830.062239] LustreError: 163107:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3830.065427] LustreError: 163107:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3830.137529] Lustre: server umount lustre-OST0000 complete [ 3832.804038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3837.823863] Lustre: server umount lustre-MDT0000 complete [ 3837.922275] LustreError: 158818:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3837.931245] LustreError: 158818:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3839.643647] LustreError: 158798:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740832 with bad export cookie 13217371085009362567 [ 3839.646975] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3839.651185] LustreError: 158798:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3839.832863] Lustre: server umount lustre-MDT0001 complete [ 3844.741841] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 3846.140147] Key type lgssc unregistered [ 3846.297545] LNet: 164042:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3846.301305] LNetError: 164042:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3846.309470] LNet: Removed LNI 192.168.204.140@tcp [ 3846.681701] Key type .llcrypt unregistered [ 3846.683539] Key type ._llcrypt unregistered [ 3869.977423] Key type ._llcrypt registered [ 3869.979219] Key type .llcrypt registered [ 3870.036562] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3870.661856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3870.675957] alg: No test for adler32 (adler32-zlib) [ 3871.568528] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 3871.684419] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 3873.295248] Key type lgssc registered [ 3873.893563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3877.671340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3877.684076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3878.940383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3880.772443] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3884.891152] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3887.277605] LustreError: 165567:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3887.283147] LustreError: 165567:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3887.287637] LustreError: 165567:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 3887.363294] LustreError: 166090:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3887.462818] Lustre: server umount lustre-MDT0000 complete [ 3894.236505] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 3898.538269] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 3903.042106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3906.206405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3908.902292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3911.749500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3914.679636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3914.712532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3914.826376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3914.848411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3914.889128] Lustre: lustre-MDT0000: new disk, initializing [ 3914.926311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3914.937984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3916.710626] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3921.948252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3921.971623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3922.010447] Lustre: 168732:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3922.040123] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3922.042587] Lustre: Skipped 1 previous similar message [ 3922.078828] Lustre: lustre-MDT0001: new disk, initializing [ 3922.115572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3922.131118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3922.135127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3923.784331] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3927.665732] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3929.355477] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3932.686506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3932.738496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3932.869785] Lustre: lustre-OST0000: new disk, initializing [ 3932.872964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3932.918596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3935.326233] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 3935.615335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3935.619062] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3935.657700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3939.019248] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3941.009032] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3941.677534] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 15:08:53 (1758740933) [ 4032.992436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4032.992820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4032.997617] Lustre: Skipped 2 previous similar messages [ 4032.999871] Lustre: Skipped 1 previous similar message [ 4038.111607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4038.114704] Lustre: Skipped 2 previous similar messages [ 4038.587518] LustreError: 172237:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4038.589706] LustreError: 172237:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4038.640359] Lustre: server umount lustre-MDT0000 complete [ 4039.909754] LustreError: 169521:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741032 with bad export cookie 13535796054948745831 [ 4039.910641] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4039.918161] LustreError: 169521:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4043.231489] LustreError: 171087:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4043.231567] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4043.239125] LustreError: 171087:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4043.243564] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4044.255631] LustreError: 170453:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4046.013384] LustreError: 172439:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4046.015983] LustreError: 172439:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4046.093882] Lustre: server umount lustre-MDT0001 complete [ 4053.608865] LustreError: 172640:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4053.611929] LustreError: 172640:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4053.677535] Lustre: server umount lustre-OST0000 complete [ 4057.267726] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 4060.637405] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4064.711951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4067.638247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4070.391927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4073.190354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4076.245830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4076.273939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4076.404274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4076.420648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4076.463410] Lustre: lustre-MDT0000: new disk, initializing [ 4076.505541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4076.515205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4078.174585] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4083.088342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4083.113191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4083.152975] Lustre: 174976:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4083.180463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4083.183276] Lustre: Skipped 1 previous similar message [ 4083.223864] Lustre: lustre-MDT0001: new disk, initializing [ 4083.278903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4083.283124] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4084.821800] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4088.306796] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4089.707094] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4092.426342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4092.460581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4092.577417] Lustre: lustre-OST0000: new disk, initializing [ 4092.579847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4092.620990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4092.623883] Lustre: Skipped 1 previous similar message [ 4095.015869] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4098.677465] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4100.341516] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4100.603434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4100.608130] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4100.653328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4101.473422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4103.106263] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4103.204797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4105.697453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4105.699888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4105.700609] LustreError: Skipped 1 previous similar message [ 4105.701286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4105.705773] Lustre: Skipped 1 previous similar message [ 4105.707554] Lustre: Skipped 1 previous similar message [ 4106.028112] LustreError: 177921:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4106.031138] LustreError: 177921:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4106.106598] Lustre: server umount lustre-OST0000 complete [ 4108.768083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4108.771828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4110.816461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4110.818980] Lustre: Skipped 3 previous similar messages [ 4113.749511] Lustre: server umount lustre-MDT0000 complete [ 4114.996171] LustreError: 175756:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741107 with bad export cookie 13535796054948904661 [ 4114.999088] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4115.001442] LustreError: 175756:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4115.072408] LustreError: 178323:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4115.075396] LustreError: 178323:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4115.165295] Lustre: server umount lustre-MDT0001 complete [ 4118.042644] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:11:49 (1758741109) [ 4123.913706] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 4127.691140] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4131.992041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4134.861862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4137.589145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4140.287167] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4143.174842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4143.208642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4143.320161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4143.336174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4143.377939] Lustre: lustre-MDT0000: new disk, initializing [ 4143.414293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4143.425154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4145.077671] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4150.047384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4150.078198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4150.130800] Lustre: 181308:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4150.264116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4151.819357] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4155.488663] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4156.891927] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4159.329596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4159.352796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4161.659450] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4165.244929] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4168.705036] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4168.755326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4173.794553] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4173.798920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4173.799030] LustreError: Skipped 1 previous similar message [ 4173.804676] Lustre: Skipped 1 previous similar message [ 4173.807518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4173.811403] Lustre: Skipped 1 previous similar message [ 4174.956109] LustreError: 183857:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4174.959016] LustreError: 183857:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4175.045820] Lustre: server umount lustre-OST0000 complete [ 4178.592282] Lustre: server umount lustre-MDT0000 complete [ 4178.913497] LustreError: 183071:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4178.920789] LustreError: 183071:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4180.119469] LustreError: 182094:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741172 with bad export cookie 13535796054948905529 [ 4180.122487] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4180.124179] LustreError: 182094:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4180.291726] Lustre: server umount lustre-MDT0001 complete [ 4183.571643] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:12:55 (1758741175) [ 4190.128616] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 4193.856243] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4197.877258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4200.797583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4203.323828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4206.061462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4208.970960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4209.024616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4209.149885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4209.168251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4209.171051] Lustre: Skipped 3 previous similar messages [ 4209.215663] Lustre: lustre-MDT0000: new disk, initializing [ 4209.217536] Lustre: Skipped 2 previous similar messages [ 4209.267130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4209.270932] Lustre: Skipped 2 previous similar messages [ 4209.280141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4209.284613] Lustre: Skipped 2 previous similar messages [ 4211.026263] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4215.801704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4215.828701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4215.873057] Lustre: 187245:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4215.998630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4217.553677] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4221.016163] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4222.521180] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4225.295474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4225.327413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4227.774405] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4231.204146] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4231.296891] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4236.257408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4236.259378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4236.263661] LustreError: Skipped 1 previous similar message [ 4236.270697] Lustre: Skipped 2 previous similar messages [ 4236.273581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4236.278385] Lustre: Skipped 1 previous similar message [ 4239.531097] LustreError: 189790:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4239.533938] LustreError: 189790:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4239.603278] Lustre: server umount lustre-OST0000 complete [ 4247.199185] Lustre: server umount lustre-MDT0000 complete [ 4248.624356] LustreError: 188029:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741241 with bad export cookie 13535796054948906614 [ 4248.627105] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4248.630104] LustreError: 188029:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4248.795133] Lustre: server umount lustre-MDT0001 complete [ 4252.866305] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 4254.061634] Key type lgssc unregistered [ 4254.225510] LNet: 190723:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4254.229193] LNetError: 190723:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4254.238831] LNet: Removed LNI 192.168.204.140@tcp [ 4254.614277] Key type .llcrypt unregistered [ 4254.615708] Key type ._llcrypt unregistered [ 4265.145200] Key type ._llcrypt registered [ 4265.146823] Key type .llcrypt registered [ 4265.192181] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4265.553987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4265.585967] alg: No test for adler32 (adler32-zlib) [ 4266.460728] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 4266.563647] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 4268.151355] Key type lgssc registered [ 4268.624704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4271.833298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4271.846052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4273.047060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4274.706517] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4278.514081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4278.691064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4280.317767] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4282.529913] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4283.986863] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4286.731755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4286.914250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4289.258190] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4291.509910] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4298.220297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4314.592418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4314.595835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4314.601630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4318.693150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4318.694091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4318.694127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4318.698774] Lustre: Skipped 1 previous similar message [ 4319.987141] LustreError: 194364:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4320.078423] Lustre: server umount lustre-OST0000 complete [ 4323.813285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4323.815502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4323.820120] Lustre: Skipped 1 previous similar message [ 4323.825641] Lustre: Skipped 1 previous similar message [ 4328.928870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4328.932146] Lustre: Skipped 1 previous similar message [ 4334.048609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4334.051168] Lustre: Skipped 1 previous similar message [ 4336.095217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4336.137379] LustreError: 194566:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4336.140518] LustreError: 194566:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4336.180860] Lustre: server umount lustre-MDT0000 complete [ 4337.808630] LustreError: 192076:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741330 with bad export cookie 372421005225864695 [ 4337.813483] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4337.814755] LustreError: 192076:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4337.893870] LustreError: 194768:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4337.896866] LustreError: 194768:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4338.024989] Lustre: server umount lustre-MDT0001 complete [ 4342.513304] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 4343.947664] Key type lgssc unregistered [ 4344.117795] LNet: 195299:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4344.122113] LNetError: 195299:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4344.129524] LNet: Removed LNI 192.168.204.140@tcp [ 4344.510304] Key type .llcrypt unregistered [ 4344.511836] Key type ._llcrypt unregistered [ 4356.038940] Key type ._llcrypt registered [ 4356.040778] Key type .llcrypt registered [ 4356.095556] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4356.537865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4356.570680] alg: No test for adler32 (adler32-zlib) [ 4357.448930] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 4357.555937] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 4359.151232] Key type lgssc registered [ 4359.666340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4363.164842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4363.177618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4364.397700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4366.066909] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4369.741903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4369.911550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4371.625585] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4374.058501] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4375.680582] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4378.551477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4378.717622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4381.280310] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4383.711484] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4388.966576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4394.468766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4394.468872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4394.473613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4394.476818] Lustre: Skipped 1 previous similar message [ 4394.481147] Lustre: Skipped 1 previous similar message [ 4395.109910] Lustre: lustre-OST0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 4398.259176] LustreError: 198919:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4398.336799] Lustre: server umount lustre-OST0000 complete [ 4400.464238] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8af44795e000.ost_server_uuid 50 [ 4401.131204] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4411.323595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4411.324482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4411.329929] Lustre: Skipped 1 previous similar message [ 4411.333957] Lustre: Skipped 1 previous similar message [ 4414.432957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4414.435896] Lustre: Skipped 1 previous similar message [ 4417.476191] LustreError: 199266:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4417.478892] LustreError: 199266:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4417.514808] Lustre: server umount lustre-MDT0000 complete [ 4419.016377] LustreError: 199267:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741411 with bad export cookie 14433943652677865792 [ 4419.019407] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4419.022021] LustreError: 199267:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4419.084535] LustreError: 199470:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4419.087490] LustreError: 199470:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4419.179333] Lustre: server umount lustre-MDT0001 complete [ 4422.831352] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 15:16:54 (1758741414) [ 4425.565458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4425.766592] LustreError: 200140:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4425.775040] LustreError: 200140:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4425.799470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4427.383397] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4430.815882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4430.816138] LustreError: 200140:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4430.979406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4432.687860] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4434.961975] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4436.415396] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4439.097992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4439.262671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4441.632322] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4443.748351] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4446.577897] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4446.609555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4446.691878] Lustre: lustre-OST0001: new disk, initializing [ 4446.694941] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4449.105903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4449.110710] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4449.172781] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4449.193282] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4451.176632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4452.863974] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4456.419720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4456.420313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4456.421645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4456.430662] Lustre: Skipped 1 previous similar message [ 4460.014128] LustreError: 203342:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4460.017131] LustreError: 203342:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4460.036480] Lustre: server umount lustre-OST0000 complete [ 4461.537156] LustreError: 201735:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4461.544994] LustreError: 201735:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4462.347464] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4462.433113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4463.703240] LustreError: 203525:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4463.944162] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4464.034259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4468.821656] LustreError: 203525:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4468.828386] LustreError: 203525:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4474.848357] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4474.851961] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4474.858149] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4474.860786] Lustre: Skipped 3 previous similar messages [ 4476.897284] LustreError: 201733:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4476.902922] LustreError: 201733:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 4480.238019] LustreError: 204044:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4480.240037] LustreError: 204044:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4480.312641] Lustre: server umount lustre-OST0001 complete [ 4482.016996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4482.021596] Lustre: Skipped 1 previous similar message [ 4492.255955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4492.258862] Lustre: Skipped 7 previous similar messages [ 4496.351194] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4496.395320] LustreError: 204245:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4496.397420] LustreError: 204245:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4496.431073] Lustre: server umount lustre-MDT0000 complete [ 4497.377333] LustreError: 200140:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4497.383855] LustreError: 200140:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4497.909412] LustreError: 200119:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741490 with bad export cookie 14433943652677866849 [ 4497.912647] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4497.914249] LustreError: 200119:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4498.081758] Lustre: server umount lustre-MDT0001 complete [ 4500.327241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4501.779370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4503.293134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4504.794839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4507.182305] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 15:18:18 (1758741498) [ 4509.689777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4509.795530] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4509.811777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4509.883661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4509.886139] Lustre: Skipped 1 previous similar message [ 4511.321508] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4514.628127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4514.663820] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4514.680751] Lustre: 205940:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4516.369304] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4518.536392] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4520.002914] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4522.643408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4522.735157] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4524.987290] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4527.181417] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4527.644643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 4529.886525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4529.931155] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4532.520687] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4533.212529] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4534.659877] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4537.824390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4537.824505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4537.827660] LustreError: Skipped 2 previous similar messages [ 4537.833147] Lustre: Skipped 1 previous similar message [ 4537.835925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4537.839590] Lustre: Skipped 1 previous similar message [ 4550.111245] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4550.129149] LustreError: 209135:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4550.131921] LustreError: 209135:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4550.154398] Lustre: server umount lustre-OST0000 complete [ 4550.232922] LustreError: 207568:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4563.938195] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4563.938937] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4563.941289] LustreError: Skipped 1 previous similar message [ 4563.948127] Lustre: Skipped 1 previous similar message [ 4567.032818] Lustre: server umount lustre-OST0001 complete [ 4574.694233] Lustre: server umount lustre-MDT0000 complete [ 4576.037210] LustreError: 207587:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741568 with bad export cookie 14433943652677868235 [ 4576.040356] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4576.041846] LustreError: 207587:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4576.190950] Lustre: server umount lustre-MDT0001 complete [ 4578.237817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4579.692333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4581.030387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4582.458712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4584.809636] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 15:19:36 (1758741576) [ 4590.120512] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 4593.387453] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 4596.987899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4599.450043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4601.595577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4603.828096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4606.298464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4606.331780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4606.413406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4606.424967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4606.459830] Lustre: lustre-MDT0000: new disk, initializing [ 4606.484309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4606.487147] Lustre: Skipped 3 previous similar messages [ 4606.493050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4607.858781] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4612.117542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4612.142421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4612.178285] Lustre: 213559:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4612.207161] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4612.209571] Lustre: Skipped 1 previous similar message [ 4612.248808] Lustre: lustre-MDT0001: new disk, initializing [ 4612.299773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4612.302789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4613.726327] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4616.980605] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4618.259906] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4620.652371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4620.672920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4620.776564] Lustre: lustre-OST0000: new disk, initializing [ 4620.779460] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4622.797587] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4626.024265] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4627.446295] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4627.459156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4627.462537] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4627.499822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4628.585291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4630.082635] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4630.181136] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4632.546593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4632.547478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4632.548824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4632.548832] Lustre: Skipped 14 previous similar messages [ 4632.549728] LustreError: Skipped 1 previous similar message [ 4632.558179] Lustre: Skipped 3 previous similar messages [ 4636.972154] LustreError: 216502:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4636.974713] LustreError: 216502:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4637.043572] Lustre: server umount lustre-OST0000 complete [ 4644.511103] Lustre: server umount lustre-MDT0000 complete [ 4645.880515] LustreError: 213550:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741638 with bad export cookie 14433943652677870006 [ 4645.884929] LustreError: 213550:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4645.885275] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4646.046243] Lustre: server umount lustre-MDT0001 complete [ 4649.004299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4649.189441] LustreError: 217383:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4649.196404] LustreError: 217383:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 7 previous similar messages [ 4650.584142] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4653.598777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4655.243299] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4657.185509] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4658.431052] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4660.850426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4662.925695] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4664.837581] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4666.305628] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4674.621144] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 4676.103769] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4676.192658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4679.650673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4679.653460] LustreError: Skipped 1 previous similar message [ 4683.053438] Lustre: server umount lustre-OST0000 complete [ 4685.290159] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4686.852267] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4710.797113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4710.928540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4710.931538] Lustre: Skipped 5 previous similar messages [ 4713.056487] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4715.066058] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4715.785123] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4719.086217] LustreError: 221816:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4719.089025] LustreError: 221816:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 4719.168633] Lustre: server umount lustre-OST0000 complete [ 4722.918935] Lustre: server umount lustre-MDT0000 complete [ 4724.359921] LustreError: 217410:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741716 with bad export cookie 14433943652677870874 [ 4724.364411] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4724.364742] LustreError: 217410:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4724.550354] Lustre: server umount lustre-MDT0001 complete [ 4728.055580] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:21:59 (1758741719) [ 4730.625031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4732.322843] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4735.488216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4737.074581] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4739.112301] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4740.605174] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4743.107477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4745.396662] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4747.570351] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4749.111373] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4757.396651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 4758.764735] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4758.857295] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4761.241112] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4761.271680] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4761.327510] Lustre: lustre-OST0001: new disk, initializing [ 4761.330490] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4763.383409] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4766.660995] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4767.756843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4767.759485] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4767.803632] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4768.074451] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4768.147605] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4769.453511] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4769.532915] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4772.836283] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4772.836301] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4772.836955] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4772.836960] Lustre: Skipped 5 previous similar messages [ 4772.839155] LustreError: Skipped 1 previous similar message [ 4772.852210] Lustre: Skipped 5 previous similar messages [ 4776.250609] Lustre: server umount lustre-OST0001 complete [ 4777.958836] LustreError: 224487:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4777.965333] LustreError: 224487:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4778.541940] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4778.629945] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4780.198862] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4780.311608] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4804.275577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4806.538305] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4808.661708] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4809.407362] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4809.703921] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4809.709819] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4809.709887] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4809.719553] Lustre: Skipped 1 previous similar message [ 4812.415557] Lustre: server umount lustre-OST0001 complete [ 4839.391193] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4839.479430] Lustre: server umount lustre-OST0000 complete [ 4843.488320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4843.492800] LustreError: Skipped 2 previous similar messages [ 4847.382979] LustreError: 229226:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4847.385876] LustreError: 229226:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 4847.443942] Lustre: server umount lustre-MDT0000 complete [ 4849.033847] LustreError: 222867:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741841 with bad export cookie 14433943652677871805 [ 4849.038728] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4849.039026] LustreError: 222867:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4855.278322] Lustre: server umount lustre-MDT0001 complete [ 4857.518458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4859.193050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4860.749881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4862.279126] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4865.004250] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 15:24:16 (1758741856) [ 4867.779988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4867.926910] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4867.939342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4868.025501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4868.028088] Lustre: Skipped 5 previous similar messages [ 4869.447633] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4872.615784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4872.654148] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4872.666394] Lustre: 230919:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4874.254948] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4876.296864] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4877.584326] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4880.030343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4880.101732] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4882.338372] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4884.322810] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4893.806212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4893.840711] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4896.271567] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4898.389348] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4902.086134] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 4903.641253] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4903.744666] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4905.336545] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4905.442788] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4906.928236] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8af44b6f7000.ost_server_uuid 50 [ 4907.515169] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8af44b6f7000.ost_server_uuid in FULL state after 0 sec [ 4907.921608] Lustre: Permanently deactivating lustre-OST0001 [ 4907.928550] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4908.891328] Lustre: Permanently reactivating lustre-OST0001 [ 4912.103200] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4912.106958] Lustre: Skipped 7 previous similar messages [ 4916.091426] Lustre: server umount lustre-OST0001 complete [ 4923.564580] Lustre: server umount lustre-OST0000 complete [ 4939.231162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4939.300708] Lustre: server umount lustre-MDT0000 complete [ 4940.729118] LustreError: 230912:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741933 with bad export cookie 14433943652677873079 [ 4940.733117] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4940.734320] LustreError: 230912:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4940.898142] Lustre: server umount lustre-MDT0001 complete [ 4943.094924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4944.567322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4945.957077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4947.347782] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4949.896927] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 15:25:41 (1758741941) [ 4950.277891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4952.702941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4952.807446] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4952.819944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4952.824508] Lustre: Skipped 5 previous similar messages [ 4954.384238] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4957.623431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4957.676262] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4957.689038] Lustre: 237062:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4959.311800] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4961.218973] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4962.439703] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4964.714550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4964.738081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4964.808663] Lustre: Permanently deactivating lustre-OST0000 [ 4964.813868] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4966.737493] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4968.648732] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4970.481581] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4971.317215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4973.473681] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 4975.389548] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4977.175266] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4981.282624] Lustre: Permanently reactivating lustre-OST0000 [ 4981.289138] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 4981.291577] Lustre: Skipped 2 previous similar messages [ 4983.225108] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4983.254876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4987.365191] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4987.368973] LustreError: Skipped 4 previous similar messages [ 4990.399141] Lustre: server umount lustre-OST0001 complete [ 5006.303353] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5006.385167] Lustre: server umount lustre-OST0000 complete [ 5022.175212] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5022.243149] Lustre: server umount lustre-MDT0000 complete [ 5023.734403] LustreError: 237788:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758742016 with bad export cookie 14433943652677874682 [ 5023.738032] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5023.739434] LustreError: 237788:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5023.917867] Lustre: server umount lustre-MDT0001 complete [ 5027.927217] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5029.086746] Key type lgssc unregistered [ 5029.230539] LNet: 241459:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5029.233385] LNetError: 241459:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5029.246785] LNet: Removed LNI 192.168.204.140@tcp [ 5029.601963] Key type .llcrypt unregistered [ 5029.603537] Key type ._llcrypt unregistered [ 5038.408613] Key type ._llcrypt registered [ 5038.409876] Key type .llcrypt registered [ 5038.447252] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5038.939281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5038.946960] alg: No test for adler32 (adler32-zlib) [ 5039.820051] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 5039.913202] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 5041.495284] Key type lgssc registered [ 5042.002402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5043.449228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5046.220398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5046.229702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5047.424139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5049.175778] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5053.091880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5053.128354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5053.177610] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5053.192155] Lustre: 242874:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5053.198550] Lustre: Permanently deactivating lustre-MDT0001 [ 5053.204584] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5053.233976] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5053.236779] LustreError: 242916:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5053.241330] LustreError: 242916:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5053.247049] LustreError: 242916:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 5053.341848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5055.027384] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5057.329723] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5058.789762] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5061.571083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5061.745235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5062.756095] LustreError: 244506:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5062.766563] LustreError: 244506:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 5064.251415] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5066.698256] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5068.259695] LustreError: 244641:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5070.329236] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5070.445987] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5071.423732] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5072.490635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5072.496858] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5073.041777] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5075.618385] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5077.376423] Lustre: Permanently reactivating lustre-MDT0001 [ 5077.384379] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5077.387585] Lustre: Skipped 2 previous similar messages [ 5084.768870] Lustre: lustre-MDT0001: Client d14e962f-0b5e-4948-a8ae-3df4e64f4bf3 (at 192.168.204.40@tcp) reconnecting [ 5087.188754] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5087.937576] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5089.950859] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5090.039914] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5091.534450] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5091.627518] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5092.679517] Lustre: Permanently deactivating lustre-MDT0001 [ 5092.688823] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5092.692221] Lustre: Skipped 2 previous similar messages [ 5100.548361] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5108.706050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5108.707336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5108.712056] Lustre: Skipped 3 previous similar messages [ 5108.714855] Lustre: Skipped 2 previous similar messages [ 5113.162666] LustreError: 247048:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5113.196152] Lustre: server umount lustre-MDT0000 complete [ 5113.824386] LustreError: 242886:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5113.831889] LustreError: 242886:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 5114.634955] LustreError: 242865:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758742107 with bad export cookie 15323532460755125324 [ 5114.638403] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5114.639694] LustreError: 242865:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5114.708746] LustreError: 247249:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5114.710835] LustreError: 247249:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5114.815442] Lustre: server umount lustre-MDT0001 complete [ 5122.348080] LustreError: 247451:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5122.350021] LustreError: 247451:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5122.366810] Lustre: server umount lustre-OST0000 complete [ 5126.395844] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 15:28:38 (1758742118) [ 5130.403520] LustreError: 248457:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5130.405919] LustreError: 248457:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5130.481555] Lustre: server umount lustre-OST0001 complete [ 5132.904648] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 5136.786362] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5140.720538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5143.462356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5146.121376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5148.821726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5151.571186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5151.589645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5151.675624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5151.679521] Lustre: Skipped 2 previous similar messages [ 5151.690048] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5151.728348] Lustre: lustre-MDT0000: new disk, initializing [ 5151.755990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5151.763627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5153.194288] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5157.543296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5157.572035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5157.612329] Lustre: 250638:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5157.639906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5157.642395] Lustre: Skipped 1 previous similar message [ 5157.678881] Lustre: lustre-MDT0001: new disk, initializing [ 5157.727208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5157.731032] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5159.180549] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5162.668778] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5164.176644] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5166.814485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5166.842776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5166.940083] Lustre: lustre-OST0000: new disk, initializing [ 5166.942715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5166.971839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5166.974096] Lustre: Skipped 1 previous similar message [ 5168.440941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5168.445095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5168.481770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5169.014166] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5172.404812] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5173.170757] LustreError: 250644:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5177.642298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5177.664613] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5177.731604] Lustre: lustre-OST0001: new disk, initializing [ 5177.734212] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5179.915608] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5183.194602] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5183.263114] LustreError: 250644:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5183.274294] LustreError: 250644:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5183.947711] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5186.575035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5186.579554] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5186.613252] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5193.367112] LustreError: 250644:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5196.777615] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5196.777848] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5196.779240] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5196.780420] LustreError: Skipped 1 previous similar message [ 5196.783360] Lustre: Skipped 1 previous similar message [ 5200.177102] LustreError: 254131:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5200.180144] LustreError: 254131:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5200.203466] Lustre: server umount lustre-OST0001 complete [ 5201.891563] LustreError: 252369:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5201.897629] LustreError: 252369:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5207.011431] LustreError: 252367:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5207.017565] LustreError: 252367:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5214.688483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5214.688541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5214.692363] LustreError: Skipped 1 previous similar message [ 5214.698026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5214.701418] Lustre: Skipped 3 previous similar messages [ 5217.251404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5217.253096] LustreError: 252369:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5217.254447] Lustre: Skipped 1 previous similar message [ 5217.262445] LustreError: 252369:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 5218.669813] LustreError: 254355:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5218.672711] LustreError: 254355:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5218.743289] Lustre: server umount lustre-OST0000 complete [ 5222.368920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5222.372344] Lustre: Skipped 1 previous similar message [ 5222.374860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5222.377540] Lustre: Skipped 1 previous similar message [ 5226.338648] Lustre: server umount lustre-MDT0000 complete [ 5227.906684] LustreError: 250629:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758742220 with bad export cookie 15323532460755127921 [ 5227.911369] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5227.912454] LustreError: 250629:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5242.335144] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5242.384043] LustreError: 254760:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5242.386165] LustreError: 254760:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5242.496541] Lustre: server umount lustre-MDT0001 complete [ 5247.033418] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5248.288394] Key type lgssc unregistered [ 5248.419522] LNet: 255293:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5248.422349] LNetError: 255293:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5248.433358] LNet: Removed LNI 192.168.204.140@tcp [ 5248.782461] Key type .llcrypt unregistered [ 5248.783897] Key type ._llcrypt unregistered [ 5260.939420] Key type ._llcrypt registered [ 5260.940701] Key type .llcrypt registered [ 5260.982840] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 5268.016884] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5268.445635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5268.461357] alg: No test for adler32 (adler32-zlib) [ 5269.339044] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 5269.438103] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 5271.031161] Key type lgssc registered [ 5271.530621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5275.255515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5278.026896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5280.481687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5283.098565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5285.990408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5286.019877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5286.030677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5287.144618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5287.161892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5287.202392] Lustre: lustre-MDT0000: new disk, initializing [ 5287.244501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5287.253348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5288.831687] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5293.442611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5293.470857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5293.510577] Lustre: 259216:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5293.535170] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5293.538196] Lustre: Skipped 1 previous similar message [ 5293.576364] Lustre: lustre-MDT0001: new disk, initializing [ 5293.609492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5293.622427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5293.625660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5295.029628] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5298.208148] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5299.510624] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5301.956931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5301.980183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5302.081656] Lustre: lustre-OST0000: new disk, initializing [ 5302.084616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5302.120576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5304.089668] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5307.150142] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5308.492094] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5311.991862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5311.996077] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5312.029112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5312.681723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 5314.094470] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5314.186590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5317.089685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5317.090057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5317.092192] LustreError: Skipped 1 previous similar message [ 5317.092920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5317.095785] Lustre: Skipped 1 previous similar message [ 5320.878658] LustreError: 262183:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5320.946171] Lustre: server umount lustre-OST0000 complete [ 5324.255959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5324.258980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5324.263694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5324.265740] Lustre: Skipped 1 previous similar message [ 5327.329258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5327.329475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5328.384372] LustreError: 262385:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5328.386379] LustreError: 262385:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5328.418622] Lustre: server umount lustre-MDT0000 complete [ 5329.840406] LustreError: 259206:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758742322 with bad export cookie 14207366933237060147 [ 5329.844083] LustreError: 259206:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5329.844366] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5329.910515] LustreError: 262587:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5329.913096] LustreError: 262587:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5330.006454] Lustre: server umount lustre-MDT0001 complete [ 5333.215730] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 15:32:04 (1758742324) [ 5335.575909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5335.762215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5337.233935] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5340.260128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5340.380704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5341.819254] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5343.883293] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5345.364409] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5347.892456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5350.261723] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5352.405858] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5401.567953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5401.571497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5401.577932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5401.580124] Lustre: Skipped 1 previous similar message [ 5403.619680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5407.023093] LustreError: 265600:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5407.025726] LustreError: 265600:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5407.100588] Lustre: server umount lustre-OST0000 complete [ 5408.695649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5413.039548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5413.156343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5413.158711] Lustre: Skipped 1 previous similar message [ 5415.114529] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5417.113410] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5424.418467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.422146] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5429.540304] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.544152] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5455.330674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5455.331533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5455.334053] LustreError: Skipped 1 previous similar message [ 5455.338878] Lustre: Skipped 1 previous similar message [ 5455.339741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5455.344939] Lustre: Skipped 3 previous similar messages [ 5456.686173] LustreError: 267264:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5456.689063] LustreError: 267264:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5456.785493] Lustre: server umount lustre-OST0000 complete [ 5464.219755] Lustre: server umount lustre-MDT0000 complete [ 5465.508217] LustreError: 263238:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758742457 with bad export cookie 14207366933237061015 [ 5465.511865] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5465.512803] LustreError: 263238:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5465.581530] LustreError: 267668:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5465.583461] LustreError: 267668:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5465.661731] Lustre: server umount lustre-MDT0001 complete [ 5469.680319] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5470.789985] Key type lgssc unregistered [ 5470.914378] LNet: 268199:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5470.916852] LNetError: 268199:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5470.927329] LNet: Removed LNI 192.168.204.140@tcp [ 5471.242123] Key type .llcrypt unregistered [ 5471.243564] Key type ._llcrypt unregistered [ 5480.400567] Key type ._llcrypt registered [ 5480.401843] Key type .llcrypt registered [ 5480.437804] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5480.807381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5480.850226] alg: No test for adler32 (adler32-zlib) [ 5481.712671] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 5481.798423] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 5483.375178] Key type lgssc registered [ 5483.746588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5486.386860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5486.395860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5487.556317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5488.918408] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5491.962850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5492.087047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5493.332364] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5495.029193] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5496.185322] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5498.366538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5498.477460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5500.364078] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5502.209951] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5511.782558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5522.400927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5522.401930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5522.404953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5522.409200] Lustre: Skipped 1 previous similar message [ 5526.636918] LustreError: 272610:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5526.695679] Lustre: server umount lustre-OST0000 complete [ 5528.032853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5528.036060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5528.041770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5528.044199] Lustre: Skipped 1 previous similar message [ 5532.640283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5532.640652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5537.760096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5537.762254] Lustre: Skipped 2 previous similar messages [ 5541.856741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5541.858834] Lustre: Skipped 1 previous similar message [ 5542.367108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5542.398210] LustreError: 272813:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5542.400349] LustreError: 272813:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5542.424652] Lustre: server umount lustre-MDT0000 complete [ 5543.662683] LustreError: 269554:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758742536 with bad export cookie 776901560298171487 [ 5543.666363] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5543.667278] LustreError: 269554:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5543.732305] LustreError: 273015:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5543.734559] LustreError: 273015:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5543.820153] Lustre: server umount lustre-MDT0001 complete [ 5547.535725] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5548.633107] Key type lgssc unregistered [ 5548.786692] LNet: 273548:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5548.789771] LNetError: 273548:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5548.801430] LNet: Removed LNI 192.168.204.140@tcp [ 5549.151657] Key type .llcrypt unregistered [ 5549.152895] Key type ._llcrypt unregistered [ 5555.980518] Key type ._llcrypt registered [ 5555.981547] Key type .llcrypt registered [ 5556.024307] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5556.430899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5556.446666] alg: No test for adler32 (adler32-zlib) [ 5557.295629] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 5557.371749] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 5558.951134] Key type lgssc registered [ 5559.305385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5561.847813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5561.854954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5563.006611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5564.248937] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5567.157140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5567.279088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5568.661333] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5570.570760] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5571.804181] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5574.145676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5574.241459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5576.069793] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5577.835484] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5588.582053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5598.176469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5598.178834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5598.182630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5599.200755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5599.202053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5599.202303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.245693] LustreError: 277140:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5603.307021] Lustre: server umount lustre-OST0000 complete [ 5608.416134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5608.418956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5608.423584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5608.425551] Lustre: Skipped 1 previous similar message [ 5610.750213] LustreError: 277342:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5610.752951] LustreError: 277342:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5610.779841] Lustre: server umount lustre-MDT0000 complete [ 5612.005966] LustreError: 274752:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758742604 with bad export cookie 6469691121390952686 [ 5612.007957] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5612.009546] LustreError: 274752:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5612.076403] LustreError: 277543:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5612.078866] LustreError: 277543:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5612.154724] Lustre: server umount lustre-MDT0001 complete [ 5615.800953] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5616.933372] Key type lgssc unregistered [ 5617.070424] LNet: 278074:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5617.074328] LNetError: 278074:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5617.089361] LNet: Removed LNI 192.168.204.140@tcp [ 5617.410324] Key type .llcrypt unregistered [ 5617.411762] Key type ._llcrypt unregistered [ 5626.165378] Key type ._llcrypt registered [ 5626.166290] Key type .llcrypt registered [ 5626.197698] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5626.503552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5626.567080] alg: No test for adler32 (adler32-zlib) [ 5627.409136] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 5627.489516] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 5629.071163] Key type lgssc registered [ 5629.457918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5632.044860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5632.051744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5633.194310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5634.486555] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5637.347171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5637.453749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5638.647341] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5640.319952] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5641.433358] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5643.593543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5643.696584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5644.709562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5645.536837] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5647.245394] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5656.032932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5656.034960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.038792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5657.056760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5657.058060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5657.059371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5661.423136] LustreError: 282467:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5661.491452] Lustre: server umount lustre-OST0000 complete [ 5663.199948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5663.202209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5663.206661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5663.209361] Lustre: Skipped 1 previous similar message [ 5667.296295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5667.296804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5672.415949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5672.417961] Lustre: Skipped 2 previous similar messages [ 5677.023109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5677.056289] LustreError: 282669:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5677.058696] LustreError: 282669:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5677.082451] Lustre: server umount lustre-MDT0000 complete [ 5677.536083] LustreError: 279449:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5677.540689] LustreError: 279449:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 5678.237949] LustreError: 279429:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758742670 with bad export cookie 4580899653313510918 [ 5678.240808] LustreError: 279429:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5678.241515] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5678.295377] LustreError: 282871:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5678.297115] LustreError: 282871:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5678.366491] Lustre: server umount lustre-MDT0001 complete [ 5681.768499] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5682.858690] Key type lgssc unregistered [ 5682.993518] LNet: 283402:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5682.996577] LNetError: 283402:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5683.006326] LNet: Removed LNI 192.168.204.140@tcp [ 5683.307585] Key type .llcrypt unregistered [ 5683.308686] Key type ._llcrypt unregistered [ 5689.837444] Key type ._llcrypt registered [ 5689.838674] Key type .llcrypt registered [ 5689.876156] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5690.180544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5690.220197] alg: No test for adler32 (adler32-zlib) [ 5691.070673] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 5691.146035] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 5692.727129] Key type lgssc registered [ 5693.058143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5695.634414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5695.641637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5696.807815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5698.047420] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5700.818613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5700.929738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5702.246086] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5703.924429] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5705.034102] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5707.196881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5707.232898] LustreError: 286215:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5707.238033] LustreError: 286215:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 5707.303433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5709.066294] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5710.692526] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5710.713594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5715.935787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5715.938873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5715.944097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5716.960408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5716.960592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5716.962445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5721.197114] LustreError: 287007:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5721.251438] Lustre: server umount lustre-OST0000 complete [ 5726.688093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5726.690291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5726.694295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5726.696029] Lustre: Skipped 1 previous similar message [ 5728.571104] LustreError: 287209:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5728.573518] LustreError: 287209:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5728.602389] Lustre: server umount lustre-MDT0000 complete [ 5729.839103] LustreError: 284562:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758742722 with bad export cookie 16134164940987115103 [ 5729.842157] LustreError: 284562:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5729.842342] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5729.906574] LustreError: 287410:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5729.908563] LustreError: 287410:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5729.988165] Lustre: server umount lustre-MDT0001 complete [ 5733.509708] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5734.649281] Key type lgssc unregistered [ 5734.784431] LNet: 287943:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5734.787598] LNetError: 287943:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5734.795398] LNet: Removed LNI 192.168.204.140@tcp [ 5735.102959] Key type .llcrypt unregistered [ 5735.104215] Key type ._llcrypt unregistered [ 5738.379063] Key type ._llcrypt registered [ 5738.380305] Key type .llcrypt registered [ 5738.418775] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5743.637982] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 5749.710852] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5750.014210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5750.041384] alg: No test for adler32 (adler32-zlib) [ 5750.897530] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 5750.991523] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 5752.567152] Key type lgssc registered [ 5752.932869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5756.243246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5758.663764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5760.931371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5763.185709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5765.619875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5765.639589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5765.647510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5766.738251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5766.751711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5766.788333] Lustre: lustre-MDT0000: new disk, initializing [ 5766.815689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5766.823289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5768.132344] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5772.347662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5772.370333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5772.404198] Lustre: 291424:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5772.437563] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5772.438472] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5772.441749] Lustre: Skipped 1 previous similar message [ 5772.474636] Lustre: lustre-MDT0001: new disk, initializing [ 5772.505648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5772.519915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5772.523164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5773.815219] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5776.972967] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5778.208031] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5780.458072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5780.482236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5780.578103] Lustre: lustre-OST0000: new disk, initializing [ 5780.580861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5780.608552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5782.535025] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5785.609319] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5786.104550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5786.107850] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5786.136773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5787.004650] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5787.086966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5788.439940] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5788.518429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5791.201508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5791.202499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5791.203213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5791.203711] LustreError: Skipped 1 previous similar message [ 5791.206955] Lustre: Skipped 1 previous similar message [ 5795.246131] LustreError: 294364:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5795.307824] Lustre: server umount lustre-OST0000 complete [ 5798.367804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5798.370254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5798.374079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5798.375658] Lustre: Skipped 1 previous similar message [ 5801.439997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5801.440585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5802.683164] LustreError: 294565:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5802.685049] LustreError: 294565:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5802.712733] Lustre: server umount lustre-MDT0000 complete [ 5803.981658] LustreError: 291414:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758742796 with bad export cookie 6059493779385557583 [ 5803.985371] LustreError: 291414:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5803.985860] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5804.058189] LustreError: 294768:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5804.060728] LustreError: 294768:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5804.155354] Lustre: server umount lustre-MDT0001 complete [ 5806.912251] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 15:39:58 (1758742798) [ 5808.994937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5809.145357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5810.383612] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5813.012309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5814.289051] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5816.022126] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5817.124680] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5819.392771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5819.511138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5819.514368] Lustre: Skipped 1 previous similar message [ 5821.491130] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5823.376128] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5844.449911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5844.450181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5844.452477] LustreError: Skipped 1 previous similar message [ 5844.456603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5844.459726] Lustre: Skipped 2 previous similar messages [ 5847.659121] LustreError: 297705:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5847.660860] LustreError: 297705:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5847.717587] Lustre: server umount lustre-OST0000 complete [ 5849.055695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5849.058679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5849.063948] Lustre: Skipped 1 previous similar message [ 5854.688097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5854.690662] Lustre: Skipped 3 previous similar messages [ 5863.391105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5863.421404] LustreError: 297907:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5863.423257] LustreError: 297907:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5863.446683] Lustre: server umount lustre-MDT0000 complete [ 5863.904069] LustreError: 295436:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5863.908230] LustreError: 295436:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 5864.649201] LustreError: 295416:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758742857 with bad export cookie 6059493779385558465 [ 5864.651860] LustreError: 296155:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 5864.652223] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5864.653097] LustreError: 295416:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5864.813764] Lustre: server umount lustre-MDT0001 complete [ 5868.342622] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 5869.455462] Key type lgssc unregistered [ 5869.591738] LNet: 298640:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5869.595478] LNetError: 298640:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5869.607544] LNet: Removed LNI 192.168.204.140@tcp [ 5869.925667] Key type .llcrypt unregistered [ 5869.927053] Key type ._llcrypt unregistered [ 5878.865829] Key type ._llcrypt registered [ 5878.866976] Key type .llcrypt registered [ 5878.904895] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 5885.041193] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5885.397292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5885.435234] alg: No test for adler32 (adler32-zlib) [ 5886.290670] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 5886.376176] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 5887.959194] Key type lgssc registered [ 5888.329730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5891.402115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5893.635598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5895.650376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5897.784861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5899.509563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5901.324627] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5903.537228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5903.556465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5903.565042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5904.647285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5904.660599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5904.697343] Lustre: lustre-MDT0000: new disk, initializing [ 5904.724885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5904.731524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5906.010206] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5910.084269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5910.105503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5910.132574] Lustre: 302433:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5910.155155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5910.157642] Lustre: Skipped 1 previous similar message [ 5910.195496] Lustre: lustre-MDT0001: new disk, initializing [ 5910.229222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5910.242523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5910.246554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5911.508120] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5914.601143] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5915.763648] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5917.940880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5917.963131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5918.048465] Lustre: lustre-OST2710: new disk, initializing [ 5918.050142] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5918.075076] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5918.706889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5918.709971] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5918.749535] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5919.980673] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5923.113074] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5925.476717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5925.503775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5925.562802] Lustre: lustre-OST03e8: new disk, initializing [ 5925.565443] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5925.593193] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5927.585355] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5927.688774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5927.692393] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5927.729465] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5929.460184] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 5931.961485] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5932.026709] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5933.318697] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5933.404391] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5934.703513] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5934.776102] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5936.098937] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5936.176903] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5938.145570] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5938.147740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5938.149545] Lustre: Skipped 3 previous similar messages [ 5938.151324] Lustre: Skipped 2 previous similar messages [ 5943.266222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5943.268332] Lustre: Skipped 4 previous similar messages [ 5943.290204] LustreError: 306772:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5943.318917] Lustre: server umount lustre-MDT0000 complete [ 5944.527277] LustreError: 303213:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758742936 with bad export cookie 5868547363453806719 [ 5944.530895] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5944.533323] LustreError: 303213:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5944.600082] LustreError: 306974:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5944.602614] LustreError: 306974:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5944.680816] Lustre: server umount lustre-MDT0001 complete [ 5956.592119] LustreError: 307174:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5956.595231] LustreError: 307174:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5956.616093] Lustre: server umount lustre-OST2710 complete [ 5968.048297] LustreError: 307376:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5968.050556] LustreError: 307376:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5968.110735] Lustre: server umount lustre-OST03e8 complete [ 5973.691466] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 5976.671492] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 5979.955981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5982.042458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5984.015300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5986.079558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5988.535240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5988.561633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5988.643982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5988.655354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5988.691052] Lustre: lustre-MDT0000: new disk, initializing [ 5988.715977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5988.722508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5990.118344] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5994.307612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5994.336448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5994.372490] Lustre: 310167:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5994.467528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5995.879829] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 5999.065565] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6000.245811] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6002.431090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6002.459715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6002.563603] Lustre: lustre-OST0000: new disk, initializing [ 6002.564876] Lustre: Skipped 1 previous similar message [ 6002.566622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6002.568712] Lustre: Skipped 2 previous similar messages [ 6002.598037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6002.600903] Lustre: Skipped 1 previous similar message [ 6004.446506] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6005.115692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6005.118814] Lustre: Skipped 1 previous similar message [ 6005.120632] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6005.157490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6007.435554] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6008.770824] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6008.848325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6010.218540] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6010.316869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6015.456499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6015.458411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6015.459384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6015.462703] Lustre: Skipped 1 previous similar message [ 6017.068851] LustreError: 313102:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6017.071027] LustreError: 313102:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6017.128065] Lustre: server umount lustre-OST0000 complete [ 6020.063890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6020.066341] LustreError: Skipped 1 previous similar message [ 6020.067655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6020.071673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6020.074047] Lustre: Skipped 1 previous similar message [ 6024.474520] Lustre: server umount lustre-MDT0000 complete [ 6025.831753] LustreError: 310159:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758743018 with bad export cookie 5868547363453808378 [ 6025.837510] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6025.904837] LustreError: 313505:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6025.907552] LustreError: 313505:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6026.016906] Lustre: server umount lustre-MDT0001 complete [ 6029.030311] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:43:40 (1758743020) [ 6029.513307] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6030.123908] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:43:41 (1758743021) [ 6031.040899] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 6033.500733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6034.688663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6035.930409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6037.092985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6037.450183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6039.597952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6039.680832] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6039.687596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6039.739424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6041.069317] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6043.814664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6043.844288] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6043.855818] Lustre: 315632:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6045.307781] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6047.065861] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6048.247046] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6050.481768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6050.513328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6050.583589] Lustre: Denying initial registration attempt from nid 192.168.204.140@tcp, specified as failover [ 6050.588080] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6050.591625] LustreError: 317248:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 6050.593680] LustreError: 317248:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6050.595393] LustreError: 317248:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6050.642501] Lustre: server umount lustre-OST0000 complete [ 6050.643831] LustreError: 317248:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6054.368052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6054.371753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6054.375819] Lustre: Skipped 1 previous similar message [ 6054.378035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6054.381086] Lustre: Skipped 1 previous similar message [ 6065.120560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6065.123227] Lustre: Skipped 4 previous similar messages [ 6067.167123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6067.200463] LustreError: 317615:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6067.202237] LustreError: 317615:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6067.224308] Lustre: server umount lustre-MDT0000 complete [ 6068.463376] LustreError: 316357:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758743060 with bad export cookie 5868547363453809253 [ 6068.466528] LustreError: 316357:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6068.466671] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6068.589879] Lustre: server umount lustre-MDT0001 complete [ 6072.288242] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 6073.284186] Key type lgssc unregistered [ 6073.405385] LNet: 318350:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6073.408499] LNetError: 318350:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6073.417274] LNet: Removed LNI 192.168.204.140@tcp [ 6073.707460] Key type .llcrypt unregistered [ 6073.708542] Key type ._llcrypt unregistered [ 6077.243990] Key type ._llcrypt registered [ 6077.244935] Key type .llcrypt registered [ 6077.283095] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 6077.594944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6077.627680] alg: No test for adler32 (adler32-zlib) [ 6078.483565] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 6078.561935] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 6080.143156] Key type lgssc registered [ 6080.578228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6083.069489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6084.425257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6085.768586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6087.118286] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6087.472106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6094.123499] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 6097.173855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6097.182194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6098.279293] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6098.288769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6098.354564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6099.798701] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6102.801426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6102.838147] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6102.848880] Lustre: 321083:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6102.947245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6104.338740] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6106.187717] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6107.423408] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6109.772917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6109.804387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6109.873634] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6109.956899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6111.987830] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6113.912772] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6114.629963] LustreError: 323355:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6114.694420] Lustre: server umount lustre-OST0000 complete [ 6118.367978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6118.370181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6118.376332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6120.928240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6120.928863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.108850] LustreError: 323563:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 6122.110640] LustreError: 323563:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6122.132232] Lustre: server umount lustre-MDT0000 complete [ 6123.378118] LustreError: 321073:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758743115 with bad export cookie 6283090047782391115 [ 6123.381308] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6123.383108] LustreError: 321073:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6123.442461] LustreError: 323764:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6123.444642] LustreError: 323764:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6123.531088] Lustre: server umount lustre-MDT0001 complete [ 6127.301813] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 6128.411419] Key type lgssc unregistered [ 6128.556375] LNet: 324296:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6128.558705] LNetError: 324296:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6128.568466] LNet: Removed LNI 192.168.204.140@tcp [ 6128.936918] Key type .llcrypt unregistered [ 6128.938765] Key type ._llcrypt unregistered [ 6138.571840] Key type ._llcrypt registered [ 6138.572884] Key type .llcrypt registered [ 6138.612253] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 6139.059941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6139.104382] alg: No test for adler32 (adler32-zlib) [ 6139.962786] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 6140.050121] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 6141.631137] Key type lgssc registered [ 6142.014136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6144.692817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6144.700406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6145.865868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6147.260182] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6150.258226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6150.385797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6151.836938] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6153.727531] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6154.991247] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6157.411726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6157.526455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6159.540619] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6161.376708] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6179.049860] LustreError: 327939:0:(ldlm_resource.c:981:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200001b71:0x1:0x0].0xf14dc48e (00000000eb7027fc) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6181.344125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6181.346833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6181.350627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6181.980448] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 6182.369989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6182.376650] Lustre: Skipped 1 previous similar message [ 6185.147621] LustreError: 327939:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6185.175071] Lustre: server umount lustre-MDT0000 complete [ 6186.480166] LustreError: 327942:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758743178 with bad export cookie 10559844012642976302 [ 6186.483694] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6186.483900] LustreError: 327942:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6187.095204] LustreError: 325664:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6187.102085] LustreError: 325664:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 6187.102442] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.40@tcp (stopping) [ 6187.107570] Lustre: Skipped 3 previous similar messages [ 6187.488465] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6191.519879] LustreError: 327283:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6191.520222] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6191.527587] LustreError: 327283:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 6191.530028] Lustre: Skipped 1 previous similar message [ 6192.638890] LustreError: 328144:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6192.640648] LustreError: 328144:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6192.719236] Lustre: server umount lustre-MDT0001 complete [ 6194.853532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6197.832049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6211.039965] LustreError: 324980:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000008e79bd9 x1844175838405248/t0(0) o250->MGC192.168.204.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6211.176662] LustreError: 327275:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6211.185923] LustreError: 327275:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 6211.205906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6211.228798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 6212.710485] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6215.760831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6215.898636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6216.933908] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6217.340659] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6219.167525] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6221.032373] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6222.823353] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6224.057856] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6224.628985] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6226.403174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6226.403268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.404161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6226.404166] Lustre: Skipped 1 previous similar message [ 6226.412664] Lustre: Skipped 1 previous similar message [ 6236.642201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6236.644327] Lustre: Skipped 4 previous similar messages [ 6240.223136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6240.248093] LustreError: 330446:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6240.251233] LustreError: 330446:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6240.321851] Lustre: server umount lustre-OST0000 complete [ 6241.762517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6241.762917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6241.769582] Lustre: Skipped 1 previous similar message [ 6256.095144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6256.132244] LustreError: 330649:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6256.134017] LustreError: 330649:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6256.158403] Lustre: server umount lustre-MDT0000 complete [ 6257.120095] LustreError: 328953:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6257.124358] LustreError: 328953:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 6257.475362] LustreError: 325649:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758743249 with bad export cookie 10559844012642983428 [ 6257.478771] LustreError: 325649:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6257.479668] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6257.626589] Lustre: server umount lustre-MDT0001 complete [ 6261.474100] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 6262.603928] Key type lgssc unregistered [ 6262.764358] LNet: 331383:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6262.766895] LNetError: 331383:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6262.780432] LNet: Removed LNI 192.168.204.140@tcp [ 6263.115318] Key type .llcrypt unregistered [ 6263.116538] Key type ._llcrypt unregistered [ 6272.757632] Key type ._llcrypt registered [ 6272.758883] Key type .llcrypt registered [ 6272.800242] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 6273.190255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6273.244079] alg: No test for adler32 (adler32-zlib) [ 6274.110523] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 6274.200590] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 6275.775193] Key type lgssc registered [ 6276.198888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6279.138603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6279.157966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6280.346222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6281.728228] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6284.684990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6284.819206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6286.306793] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6288.152806] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6289.418924] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6294.625211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6294.625981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6294.631073] Lustre: Skipped 1 previous similar message [ 6294.634528] Lustre: Skipped 1 previous similar message [ 6296.635740] LustreError: 334157:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6296.663619] Lustre: server umount lustre-MDT0000 complete [ 6297.853301] LustreError: 332737:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758743290 with bad export cookie 152343084385002098 [ 6297.858162] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6297.918755] LustreError: 334361:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6297.920401] LustreError: 334361:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6297.994835] Lustre: server umount lustre-MDT0001 complete [ 6302.794057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6302.881654] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6302.891099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6302.957885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6304.238873] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6307.129400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6307.161105] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6307.171274] Lustre: 335130:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6307.257795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6308.523299] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6310.257771] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6311.399394] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6313.851263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6313.914275] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6313.917233] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6313.920917] LustreError: 336789:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 6313.924312] LustreError: 336789:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6313.926326] LustreError: 336789:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6313.973154] Lustre: server umount lustre-OST0000 complete [ 6313.974312] LustreError: 336789:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6316.724530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6316.789523] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6316.856574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6318.657403] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6320.383248] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6322.850115] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6322.869857] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6322.890418] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6322.915435] Lustre: lustre-OST0001: new disk, initializing [ 6322.917856] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6324.706686] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6326.081778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6326.086458] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6326.103084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 6326.122890] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6326.369642] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6331.366316] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6331.369081] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6331.370171] LustreError: Skipped 1 previous similar message [ 6331.374648] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6331.378307] Lustre: Skipped 1 previous similar message [ 6333.427109] LustreError: 338905:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6333.429040] LustreError: 338905:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6333.449451] Lustre: server umount lustre-OST0001 complete [ 6336.483201] LustreError: 337258:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6336.483546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6336.484321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6336.484328] Lustre: Skipped 1 previous similar message [ 6336.484943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6336.489300] LustreError: 337258:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 6336.492909] LustreError: Skipped 1 previous similar message [ 6341.602900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6341.602969] LustreError: 338713:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6341.604884] Lustre: Skipped 2 previous similar messages [ 6341.609902] LustreError: 338713:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 6346.722363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6346.722629] LustreError: 337258:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6346.724778] Lustre: Skipped 1 previous similar message [ 6346.731297] LustreError: 337258:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 6349.279102] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6349.297802] LustreError: 339106:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6349.300050] LustreError: 339106:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6349.364683] Lustre: server umount lustre-OST0000 complete [ 6351.841129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6351.845633] Lustre: Skipped 2 previous similar messages [ 6356.800095] LustreError: 339309:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6356.802046] LustreError: 339309:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6356.836731] Lustre: server umount lustre-MDT0000 complete [ 6356.960400] LustreError: 336548:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6356.966277] LustreError: 336548:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 6358.140540] LustreError: 335855:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758743350 with bad export cookie 152343084385002644 [ 6358.144444] LustreError: 335855:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6358.144920] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6358.310641] Lustre: server umount lustre-MDT0001 complete [ 6362.159901] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 6363.401377] Key type lgssc unregistered [ 6363.533360] LNet: 340041:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6363.535709] LNetError: 340041:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6363.545415] LNet: Removed LNI 192.168.204.140@tcp [ 6363.866892] Key type .llcrypt unregistered [ 6363.868298] Key type ._llcrypt unregistered [ 6375.578776] Key type ._llcrypt registered [ 6375.580200] Key type .llcrypt registered [ 6375.621022] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 6382.329653] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 6382.722487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6382.735321] alg: No test for adler32 (adler32-zlib) [ 6383.583346] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 6383.663296] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 6385.239167] Key type lgssc registered [ 6385.566703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6388.580553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6390.778711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6392.751025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6394.662283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6396.780143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6396.800046] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6396.805478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6397.884538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6397.894557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6397.926743] Lustre: lustre-MDT0000: new disk, initializing [ 6397.948346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6397.955661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6399.162467] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6403.313875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6403.339713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6403.380497] Lustre: 343961:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6403.409528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6403.412416] Lustre: Skipped 1 previous similar message [ 6403.457093] Lustre: lustre-MDT0001: new disk, initializing [ 6403.491796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6403.504946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6403.508416] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6404.825380] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6408.005759] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6409.262447] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6411.555832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6411.579888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6411.688035] Lustre: lustre-OST0000: new disk, initializing [ 6411.690706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6411.719314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6413.624774] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6416.381638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6416.385403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6416.424968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6416.761863] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6418.174804] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6418.260922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6419.699615] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6419.788621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6421.474415] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6421.474474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6421.477978] LustreError: Skipped 1 previous similar message [ 6421.483504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6421.487209] Lustre: Skipped 1 previous similar message [ 6422.509089] LustreError: 346900:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6422.576884] Lustre: server umount lustre-OST0000 complete [ 6424.031842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6424.034242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6424.038202] Lustre: Skipped 1 previous similar message [ 6424.040052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6426.593133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6426.593398] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6431.712576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6431.714752] Lustre: Skipped 2 previous similar messages [ 6436.832841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6436.834747] Lustre: Skipped 1 previous similar message [ 6438.367140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6438.399906] LustreError: 347101:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6438.402146] LustreError: 347101:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6438.431666] Lustre: server umount lustre-MDT0000 complete [ 6439.753403] LustreError: 344740:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758743432 with bad export cookie 14326668154689330673 [ 6439.756971] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6439.757477] LustreError: 344740:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6439.829977] LustreError: 347303:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6439.832681] LustreError: 347303:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6439.939110] Lustre: server umount lustre-MDT0001 complete [ 6442.854661] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 15:50:34 (1758743434) [ 6444.988354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6447.088725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6454.840585] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 6457.895228] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 6461.496316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6463.866829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6466.055371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6468.267602] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6470.636263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6470.660968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6470.749979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6470.762961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6470.802776] Lustre: lustre-MDT0000: new disk, initializing [ 6470.829391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6470.836792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6472.247387] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6476.562251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6476.589748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6476.625643] Lustre: 351072:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6476.654928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6476.656738] Lustre: Skipped 1 previous similar message [ 6476.695766] Lustre: lustre-MDT0001: new disk, initializing [ 6476.729273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6476.742243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6476.745696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6478.187552] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6481.420292] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6482.675367] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6485.064233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6485.091978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6485.194540] Lustre: lustre-OST0000: new disk, initializing [ 6485.197256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6485.228267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6487.162698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6487.167619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6487.187584] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6487.206840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6490.399943] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6491.851958] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6491.945499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6493.332035] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6493.406454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6497.762442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6497.763940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6497.765279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6497.766330] Lustre: Skipped 1 previous similar message [ 6497.768452] LustreError: Skipped 1 previous similar message [ 6500.139519] LustreError: 354008:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6500.141778] LustreError: 354008:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6500.218913] Lustre: server umount lustre-OST0000 complete [ 6502.368242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6502.370553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6507.707859] LustreError: 354209:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6507.709617] LustreError: 354209:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6507.744353] Lustre: server umount lustre-MDT0000 complete [ 6509.013157] LustreError: 351063:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758743501 with bad export cookie 14326668154689331541 [ 6509.016880] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6509.019926] LustreError: 351063:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6509.159016] Lustre: server umount lustre-MDT0001 complete [ 6512.029611] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:51:43 (1758743503) [ 6514.582120] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:51:46 (1758743506) [ 6517.487590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6517.655157] LustreError: 355422:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6517.660503] LustreError: 355422:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 6517.681506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6519.063721] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6522.006555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6523.434911] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6525.233602] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6526.484414] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6528.810865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6530.844124] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6532.706425] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6546.812754] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6547.356361] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6547.941872] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6548.541986] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6549.217347] Lustre: Failing over lustre-MDT0000 [ 6549.356858] LustreError: 357884:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6549.358996] LustreError: 357884:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6549.391535] Lustre: server umount lustre-MDT0000 complete [ 6553.056024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6553.058101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.061319] Lustre: Skipped 1 previous similar message [ 6553.063042] LustreError: 355440:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6558.688983] LustreError: 355440:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6558.694874] LustreError: 355440:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 6562.462405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6562.500679] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6562.593478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6562.596267] Lustre: Skipped 2 previous similar messages [ 6562.627032] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6563.926264] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6564.041592] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6567.912121] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6567.922250] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6567.939244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6569.158480] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6569.745370] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6571.639961] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6572.301909] Lustre: Failing over lustre-MDT0000 [ 6572.487983] LustreError: 358963:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6572.489853] LustreError: 358963:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6572.516217] Lustre: server umount lustre-MDT0000 complete [ 6573.026739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6573.027556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6573.028365] LustreError: 355417:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6573.028373] LustreError: 355417:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 6573.030675] Lustre: Skipped 3 previous similar messages [ 6578.145269] LustreError: 355422:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6578.149921] LustreError: 355422:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 6585.638104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6585.679642] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6585.791890] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6587.131507] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6589.024961] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6590.948734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6590.950422] Lustre: Skipped 2 previous similar messages [ 6590.956123] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6590.972442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6592.179334] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6592.737224] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6596.067383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6596.069500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6596.072558] Lustre: Skipped 5 previous similar messages [ 6601.111183] Lustre: server umount lustre-MDT0000 complete [ 6601.185207] LustreError: 358674:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6601.190075] LustreError: 358674:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 7 previous similar messages [ 6602.400311] LustreError: 355402:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758743594 with bad export cookie 14326668154689334180 [ 6602.403541] LustreError: 355402:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6602.403795] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6602.545704] Lustre: server umount lustre-MDT0001 complete [ 6614.129267] LustreError: 360395:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6614.132356] LustreError: 360395:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6614.197442] Lustre: server umount lustre-OST0000 complete [ 6619.467771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6619.623623] LustreError: 361222:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6621.010582] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6623.992303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6625.545247] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6627.454518] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6628.651624] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6630.960713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6631.074049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6631.077428] Lustre: Skipped 3 previous similar messages [ 6633.013874] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6634.782896] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6645.414982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6654.944485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6654.948028] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6654.952275] Lustre: Skipped 4 previous similar messages [ 6654.954068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6657.901286] Lustre: server umount lustre-OST0000 complete [ 6665.367228] Lustre: server umount lustre-MDT0000 complete [ 6666.208186] LustreError: 361935:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6666.214520] LustreError: 361935:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 6666.685170] LustreError: 361201:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758743659 with bad export cookie 14326668154689334908 [ 6666.688463] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6666.690349] LustreError: 361201:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6666.835897] Lustre: server umount lustre-MDT0001 complete [ 6670.769860] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 6671.892857] Key type lgssc unregistered [ 6672.032538] LNet: 364425:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6672.035790] LNetError: 364425:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6672.042343] LNet: Removed LNI 192.168.204.140@tcp [ 6672.355987] Key type .llcrypt unregistered [ 6672.357706] Key type ._llcrypt unregistered [ 6681.808297] Key type ._llcrypt registered [ 6681.809837] Key type .llcrypt registered [ 6681.851733] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 6682.253687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6682.276135] alg: No test for adler32 (adler32-zlib) [ 6683.123853] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 6683.209977] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 6684.791141] Key type lgssc registered [ 6685.161708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6689.250244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6689.262719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6690.603471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6693.854368] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6695.905296] LustreError: 365899:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6695.925942] LustreError: 365899:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 6699.946060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6700.000796] LustreError: 365899:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6700.180352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6702.631771] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6706.706925] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6709.472532] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6715.418453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6715.763692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6720.537535] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6726.211206] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6735.210345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6745.574084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6745.577310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6745.581808] Lustre: Skipped 1 previous similar message [ 6745.588115] Lustre: Skipped 1 previous similar message [ 6749.099841] LustreError: 368169:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6749.182650] Lustre: server umount lustre-MDT0000 complete [ 6750.695653] LustreError: 365899:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6750.714973] LustreError: 365899:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 6753.062176] LustreError: 365878:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758743745 with bad export cookie 17554156967738681009 [ 6753.078934] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6753.080333] LustreError: 365878:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6753.215855] LustreError: 368370:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6753.226730] LustreError: 368370:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6753.465897] Lustre: server umount lustre-MDT0001 complete [ 6767.297368] LustreError: 368571:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6767.300353] LustreError: 368571:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6767.499756] Lustre: server umount lustre-OST0000 complete [ 6779.907610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6780.428410] LustreError: 369301:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6780.599084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6784.664228] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6786.036150] LustreError: 369300:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6792.156162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6792.461839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6795.523242] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6799.240383] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6801.893887] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6806.151261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6806.354505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6809.857833] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6813.241372] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6824.810624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6838.752184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6838.755524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6838.761678] Lustre: Skipped 1 previous similar message [ 6838.764504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6840.291111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6840.293586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6840.299122] Lustre: Skipped 1 previous similar message [ 6840.306291] Lustre: Skipped 2 previous similar messages [ 6843.410530] LustreError: 371672:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6843.414371] LustreError: 371672:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6843.470329] Lustre: server umount lustre-MDT0000 complete [ 6845.410122] LustreError: 369295:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6845.417834] LustreError: 369295:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 6845.581668] LustreError: 369282:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758743837 with bad export cookie 17554156967738682122 [ 6845.585325] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6845.586678] LustreError: 369282:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6845.798736] Lustre: server umount lustre-MDT0001 complete [ 6858.237645] LustreError: 372074:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6858.243982] LustreError: 372074:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6858.419605] Lustre: server umount lustre-OST0000 complete [ 6866.779138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6867.052159] LustreError: 372902:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6867.086901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6869.019386] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6875.090093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6878.440368] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6882.242379] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6884.646647] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6889.242241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6892.529232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6893.759388] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 6897.322451] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6902.752167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6902.760666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6902.770918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6904.804054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6906.807534] LustreError: 375147:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6906.816839] LustreError: 375147:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6906.989599] Lustre: server umount lustre-OST0000 complete [ 6909.921239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6909.925716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6909.935209] Lustre: Skipped 2 previous similar messages [ 6909.942587] Lustre: Skipped 3 previous similar messages [ 6920.160051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6920.165992] Lustre: Skipped 3 previous similar messages [ 6924.255153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6924.321308] LustreError: 375348:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6924.326838] LustreError: 375348:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6924.401564] Lustre: server umount lustre-MDT0000 complete [ 6925.282121] LustreError: 373614:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6925.294103] LustreError: 373614:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 6927.186505] LustreError: 372882:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758743919 with bad export cookie 17554156967738683144 [ 6927.189083] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6927.205948] LustreError: 372882:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6927.532875] Lustre: server umount lustre-MDT0001 complete [ 6934.902330] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 6937.125249] Key type lgssc unregistered [ 6937.371264] LNet: 376083:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6937.375694] LNetError: 376083:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6937.395762] LNet: Removed LNI 192.168.204.140@tcp [ 6938.003757] Key type .llcrypt unregistered [ 6938.006408] Key type ._llcrypt unregistered [ 6954.312079] Key type ._llcrypt registered [ 6954.313506] Key type .llcrypt registered [ 6954.377551] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 6955.023424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6955.062464] alg: No test for adler32 (adler32-zlib) [ 6955.971827] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 6956.113938] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 6957.735190] Key type lgssc registered [ 6958.439644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6963.396559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6963.412273] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6963.417132] LustreError: 377481:0:(osd_handler.c:8487:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6963.433448] LustreError: 377481:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6963.439893] LustreError: 377481:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6963.444491] LustreError: 377481:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6963.452904] LustreError: 377481:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6968.638944] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6968.645724] LustreError: 377879:0:(osd_handler.c:8487:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6968.662996] LustreError: 377879:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6968.666959] LustreError: 377879:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6968.676489] LustreError: 377879:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6968.682422] LustreError: 377879:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6980.249249] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 6981.984379] Key type lgssc unregistered [ 6982.238430] LNet: 378864:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6982.247421] LNetError: 378864:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6982.259398] LNet: Removed LNI 192.168.204.140@tcp [ 6982.750905] Key type .llcrypt unregistered [ 6982.752710] Key type ._llcrypt unregistered [ 6994.306088] Key type ._llcrypt registered [ 6994.308103] Key type .llcrypt registered [ 6994.358073] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 7003.922971] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7004.419486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7004.502301] alg: No test for adler32 (adler32-zlib) [ 7005.651079] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 7005.817670] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 7007.455143] Key type lgssc registered [ 7008.350887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7013.529480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7016.726900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7019.653374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7022.821538] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7026.314355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7026.344371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7026.352824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7027.464502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7027.486200] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7027.536353] Lustre: lustre-MDT0000: new disk, initializing [ 7027.581512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7027.592553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7029.701564] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7035.274363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7035.319348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7035.363162] Lustre: 381971:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7035.393601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7035.395655] Lustre: Skipped 1 previous similar message [ 7035.436262] Lustre: lustre-MDT0001: new disk, initializing [ 7035.480136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7035.498862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7035.506271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7037.247775] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7040.974453] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7042.865669] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7046.431081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7046.478505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7046.616286] Lustre: lustre-OST0000: new disk, initializing [ 7046.618825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7046.662268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7049.723850] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7051.779855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7051.784899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7051.816440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 7054.299524] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7056.903768] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7057.017573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7059.254343] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7059.378389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7061.989119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7061.995199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7062.004319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7063.525225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7063.525406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7063.526011] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7066.616177] LustreError: 384914:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 7066.748331] Lustre: server umount lustre-OST0000 complete [ 7071.201198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7071.208588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7071.217562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7071.220480] Lustre: Skipped 1 previous similar message [ 7073.762959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7073.764022] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7075.274148] LustreError: 385126:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 7075.277025] LustreError: 385126:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7075.339875] Lustre: server umount lustre-MDT0000 complete [ 7077.890863] LustreError: 381962:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758744070 with bad export cookie 7975544295700060515 [ 7077.891994] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7077.901021] LustreError: 381962:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7078.001559] LustreError: 385327:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7078.004879] LustreError: 385327:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7078.208885] Lustre: server umount lustre-MDT0001 complete [ 7083.610720] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7084.606985] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 16:01:15 (1758744075) [ 7089.135661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7089.543763] LustreError: 386046:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7089.554954] LustreError: 386046:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 7089.607654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7092.409304] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7094.754913] LustreError: 386046:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7098.698026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7098.853164] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7098.855873] Lustre: Skipped 1 previous similar message [ 7098.979878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7101.878313] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7105.779915] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7108.266109] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7112.912989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7113.150572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7117.468313] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7121.344497] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7125.496680] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7125.553134] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7125.662513] Lustre: lustre-OST0001: new disk, initializing [ 7125.666602] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7129.689257] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7131.733863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7131.746780] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7131.793389] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7135.105937] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7141.990790] Lustre: lustre-OST0001: Not available for connect from 192.168.204.40@tcp (stopping) [ 7142.373119] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7142.373744] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7142.386500] LustreError: Skipped 1 previous similar message [ 7143.987454] LustreError: 389246:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7143.996846] LustreError: 389246:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7144.032674] Lustre: server umount lustre-OST0001 complete [ 7147.497872] LustreError: 387641:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7152.614186] LustreError: 387641:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7152.625681] LustreError: 387641:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 7157.742620] LustreError: 388452:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7157.758575] LustreError: 388452:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 7160.290708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7160.291951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7160.295691] LustreError: Skipped 1 previous similar message [ 7160.303036] Lustre: Skipped 1 previous similar message [ 7160.304246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7160.304251] Lustre: Skipped 2 previous similar messages [ 7164.470719] LustreError: 389469:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7164.477293] LustreError: 389469:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7164.590034] Lustre: server umount lustre-OST0000 complete [ 7173.199535] LustreError: 389670:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7173.205953] LustreError: 389670:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7173.285327] Lustre: server umount lustre-MDT0000 complete [ 7175.872589] LustreError: 386027:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758744168 with bad export cookie 7975544295700061383 [ 7175.880557] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7182.142479] Lustre: server umount lustre-MDT0001 complete [ 7187.720489] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 7189.448719] Key type lgssc unregistered [ 7189.654390] LNet: 390404:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7189.660552] LNetError: 390404:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7189.671683] LNet: Removed LNI 192.168.204.140@tcp [ 7190.199468] Key type .llcrypt unregistered [ 7190.201930] Key type ._llcrypt unregistered [ 7210.472780] Key type ._llcrypt registered [ 7210.474258] Key type .llcrypt registered [ 7210.560598] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 7220.672934] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7221.467351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7221.501605] alg: No test for adler32 (adler32-zlib) [ 7222.472771] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 7222.651931] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 7224.279177] Key type lgssc registered [ 7225.005833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7230.471501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7234.541229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7238.049374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7241.765391] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7245.759677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7245.802653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7245.814900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7246.982086] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7247.012643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7247.067341] Lustre: lustre-MDT0000: new disk, initializing [ 7247.136431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7247.153398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7249.595114] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7255.778723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7255.818508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7255.868358] Lustre: 394330:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7255.915826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7255.919525] Lustre: Skipped 1 previous similar message [ 7255.967461] Lustre: lustre-MDT0001: new disk, initializing [ 7256.029490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7256.056757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7256.062830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7258.057553] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7261.850239] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7263.616199] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7266.915262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7266.981883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7267.129148] Lustre: lustre-OST0000: new disk, initializing [ 7267.133090] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7267.181616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7267.796659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7267.805547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7267.884729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7270.353306] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7274.659271] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7276.595332] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7276.715923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7278.708815] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7278.821659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7283.175657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7283.179233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.188045] LustreError: Skipped 1 previous similar message [ 7283.188679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7283.209843] Lustre: Skipped 1 previous similar message [ 7285.943083] LustreError: 397270:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7286.089583] Lustre: server umount lustre-OST0000 complete [ 7291.873534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7291.878295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7291.888400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7291.893556] Lustre: Skipped 1 previous similar message [ 7293.411371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7293.413048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7294.512711] LustreError: 397471:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7294.527307] LustreError: 397471:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7294.652703] Lustre: server umount lustre-MDT0000 complete [ 7296.882228] LustreError: 395110:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758744289 with bad export cookie 10567691782074206407 [ 7296.882924] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7296.889442] LustreError: 395110:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7296.974461] LustreError: 397672:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7296.977589] LustreError: 397672:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7297.098881] Lustre: server umount lustre-MDT0001 complete [ 7301.378903] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 16:04:52 (1758744292) [ 7306.411672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7310.926040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7311.131952] Lustre: lustre-MDT0000: new disk, initializing [ 7311.212881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7313.602715] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7316.449809] LustreError: 398938:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7316.458778] LustreError: 398938:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 7318.147532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7318.353941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7320.290431] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7323.079743] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7325.011659] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7326.688982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7326.691875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7326.693932] Lustre: Skipped 1 previous similar message [ 7326.696119] Lustre: Skipped 1 previous similar message [ 7332.117120] LustreError: 400280:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7332.120444] LustreError: 400280:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7332.174771] Lustre: server umount lustre-MDT0000 complete [ 7332.835165] LustreError: 398934:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7332.851037] LustreError: 398934:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 7334.002796] LustreError: 398918:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758744326 with bad export cookie 10567691782074207296 [ 7334.005778] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7334.008049] LustreError: 398918:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7334.204192] Lustre: server umount lustre-MDT0001 complete [ 7338.492135] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 16:05:30 (1758744330) [ 7341.501736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7341.737142] LustreError: 401199:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7341.789179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7343.755517] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7347.169757] LustreError: 401199:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7348.247920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7350.522164] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7353.476905] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7355.361088] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7358.586642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7358.739589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7358.742415] Lustre: Skipped 1 previous similar message [ 7361.481534] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7364.048244] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7378.025416] Lustre: Permanently deactivating lustre-OST0000 [ 7378.032861] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7384.586281] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7386.644738] LustreError: 403913:0:(mgs_llog.c:1937:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7386.651392] LustreError: 403913:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 7386.657253] LustreError: 403913:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7388.015756] LustreError: 403962:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 7388.019758] LustreError: 403962:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7395.186098] LustreError: 404059:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7395.188181] LustreError: 404059:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7395.268363] Lustre: server umount lustre-OST0000 complete [ 7396.843522] LustreError: 404211:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 7396.847351] LustreError: 404211:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7398.882917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7398.894693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7398.898772] Lustre: Skipped 4 previous similar messages [ 7409.120871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7409.124834] Lustre: Skipped 3 previous similar messages [ 7412.191298] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7412.244532] LustreError: 404308:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7412.286296] LustreError: 404308:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7412.341884] Lustre: server umount lustre-MDT0000 complete [ 7414.241294] LustreError: 401911:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7414.248685] LustreError: 401911:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 7414.770710] LustreError: 401180:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758744407 with bad export cookie 10567691782074207835 [ 7414.776411] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7414.778468] LustreError: 401180:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7425.106796] Lustre: server umount lustre-MDT0001 complete [ 7430.337599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7432.645518] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7434.433101] LustreError: 405353:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7438.178890] Lustre: 405692:0:(mgs_llog.c:1698:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.140@tcp [ 7439.397567] LustreError: 405789:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7439.401152] LustreError: 405789:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7439.458282] Lustre: server umount MGS complete [ 7445.300062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7445.553793] LustreError: 406418:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7445.568727] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7445.571818] Lustre: Skipped 1 previous similar message [ 7445.610573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7447.946937] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7453.079773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7453.245067] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7455.793953] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7459.532089] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7462.113788] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7463.343512] Lustre: Permanently reactivating lustre-OST0000 [ 7463.357943] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7463.360545] Lustre: Skipped 2 previous similar messages [ 7490.527242] LustreError: 408010:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7494.496922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7498.032740] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7501.083980] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7510.497875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7510.499361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7510.506488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7510.510302] Lustre: Skipped 2 previous similar messages [ 7514.732442] LustreError: 409684:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7514.758083] LustreError: 409684:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7514.847211] Lustre: server umount lustre-OST0000 complete [ 7519.712659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7519.717356] LustreError: Skipped 1 previous similar message [ 7522.929781] Lustre: server umount lustre-MDT0000 complete [ 7524.683703] LustreError: 406398:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758744517 with bad export cookie 10567691782074208780 [ 7524.685212] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7524.689489] LustreError: 406398:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7524.870885] Lustre: server umount lustre-MDT0001 complete [ 7529.964855] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 7531.318910] Key type lgssc unregistered [ 7531.499183] LNet: 410625:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7531.502865] LNetError: 410625:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7531.511207] LNet: Removed LNI 192.168.204.140@tcp [ 7531.930501] Key type .llcrypt unregistered [ 7531.933319] Key type ._llcrypt unregistered [ 7540.649094] Key type ._llcrypt registered [ 7540.651511] Key type .llcrypt registered [ 7540.697238] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 7548.178540] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7548.591074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7548.632244] alg: No test for adler32 (adler32-zlib) [ 7549.536976] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 7549.638116] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 7551.231755] Key type lgssc registered [ 7551.805423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7556.188309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7559.296719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7562.281785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7565.153691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7568.485212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7568.521215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7568.530089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7569.641431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7569.660427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7569.704106] Lustre: lustre-MDT0000: new disk, initializing [ 7569.761848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7569.776711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7571.509991] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7576.698310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7576.734651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7576.782208] Lustre: 413732:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7576.811082] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7576.813626] Lustre: Skipped 1 previous similar message [ 7576.855223] Lustre: lustre-MDT0001: new disk, initializing [ 7576.901481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7576.919128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7576.924147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7578.600823] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7582.331240] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7583.997995] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7587.415398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7587.444438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7587.570858] Lustre: lustre-OST0000: new disk, initializing [ 7587.576220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7587.628131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7590.147590] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7591.442813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7591.455013] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7591.498288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7594.216778] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7596.396157] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7596.542519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7598.829511] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7598.967401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7601.634896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7601.635655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7601.640900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7601.641103] Lustre: Skipped 1 previous similar message [ 7606.070790] LustreError: 416670:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7606.181762] Lustre: server umount lustre-OST0000 complete [ 7611.873858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7611.874713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7611.879808] Lustre: Skipped 1 previous similar message [ 7611.883460] Lustre: Skipped 1 previous similar message [ 7614.282990] LustreError: 416871:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7614.285446] LustreError: 416871:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7614.329198] Lustre: server umount lustre-MDT0000 complete [ 7616.277406] LustreError: 413724:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758744608 with bad export cookie 3544434695540441391 [ 7616.281423] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7616.285105] LustreError: 413724:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7616.364374] LustreError: 417072:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7616.367560] LustreError: 417072:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7616.520614] Lustre: server umount lustre-MDT0001 complete [ 7620.530536] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7621.336106] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 16:10:12 (1758744612) [ 7624.487429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7624.760354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7626.553670] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7629.792471] LustreError: 417792:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7629.799320] LustreError: 417792:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 7630.581037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7630.782041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7632.816556] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7635.979743] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7638.190430] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7641.838198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7642.019372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7644.688874] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7647.414669] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7648.440310] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7655.608884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7665.635691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7665.639207] LustreError: Skipped 1 previous similar message [ 7665.641155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7665.648908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7665.651053] Lustre: Skipped 1 previous similar message [ 7666.663216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7671.777033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7671.779811] Lustre: Skipped 3 previous similar messages [ 7676.898076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7676.900743] Lustre: Skipped 1 previous similar message [ 7677.407511] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7677.438315] LustreError: 420248:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7677.446141] LustreError: 420248:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7677.543750] Lustre: server umount lustre-OST0000 complete [ 7682.021205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7682.021257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7682.030692] Lustre: Skipped 2 previous similar messages [ 7685.507331] LustreError: 420451:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7685.510799] LustreError: 420451:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7685.562599] Lustre: server umount lustre-MDT0000 complete [ 7687.138889] LustreError: 417792:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7687.147560] LustreError: 417792:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 7687.272239] LustreError: 417773:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758744679 with bad export cookie 3544434695540442280 [ 7687.275728] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7687.277661] LustreError: 417773:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7687.488930] Lustre: server umount lustre-MDT0001 complete [ 7692.058807] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 7693.387514] Key type lgssc unregistered [ 7693.553116] LNet: 421185:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7693.557705] LNetError: 421185:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7693.572476] LNet: Removed LNI 192.168.204.140@tcp [ 7693.954101] Key type .llcrypt unregistered [ 7693.955709] Key type ._llcrypt unregistered [ 7705.289734] Key type ._llcrypt registered [ 7705.291405] Key type .llcrypt registered [ 7705.337741] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 7705.462279] Key type .llcrypt unregistered [ 7705.464034] Key type ._llcrypt unregistered [ 7714.783773] Key type ._llcrypt registered [ 7714.785222] Key type .llcrypt registered [ 7714.865086] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7715.469753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7715.493747] alg: No test for adler32 (adler32-zlib) [ 7716.381423] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 7716.484255] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 7718.071210] Key type lgssc registered [ 7718.586664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7722.048996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7722.060211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7723.309301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7725.002709] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7729.387472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7729.582281] LustreError: 423330:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7729.588901] LustreError: 423330:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 7729.595156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7732.642763] LustreError: 423334:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7732.657037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7732.666590] LustreError: 423334:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 7732.881548] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7735.861775] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7737.824646] LustreError: 423334:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7737.853265] LustreError: 423334:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 7740.181768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7740.434552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7742.729625] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7753.186042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7753.192671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7753.206129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7755.749335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7755.750241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7755.759655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7758.639093] LustreError: 425387:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7758.735142] Lustre: server umount lustre-OST0000 complete [ 7760.864450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7760.865141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7760.880074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7760.882874] Lustre: Skipped 2 previous similar messages [ 7765.984433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7765.987399] Lustre: Skipped 1 previous similar message [ 7766.595671] LustreError: 425590:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7766.598341] LustreError: 425590:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7766.651468] Lustre: server umount lustre-MDT0000 complete [ 7768.550320] LustreError: 423316:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758744760 with bad export cookie 5422470027393062787 [ 7768.553444] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7768.556644] LustreError: 423316:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7768.647471] LustreError: 425791:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7768.651247] LustreError: 425791:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7768.790709] Lustre: server umount lustre-MDT0001 complete [ 7773.353338] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 7774.654438] Key type lgssc unregistered [ 7774.806784] LNet: 426324:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7774.810474] LNetError: 426324:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7774.817778] LNet: Removed LNI 192.168.204.140@tcp [ 7775.187538] Key type .llcrypt unregistered [ 7775.189188] Key type ._llcrypt unregistered [ 7786.675915] Key type ._llcrypt registered [ 7786.677541] Key type .llcrypt registered [ 7786.728970] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7787.244021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7787.261487] alg: No test for adler32 (adler32-zlib) [ 7788.143480] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 7788.253160] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 7789.871251] Key type lgssc registered [ 7790.372822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7794.207288] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7794.219068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7811.743360] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7817.887282] LustreError: 427670:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.140@tcp: failed processing log, type 4: rc = -110 [ 7847.583209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7850.170598] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7852.808761] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7856.072910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7856.320354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7856.354158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7858.042527] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7861.474638] LustreError: 427697:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7861.482565] LustreError: 427697:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 7861.943688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7862.130338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7863.842300] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7866.126976] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7867.699278] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7871.969562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7871.973326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7871.979949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7872.483365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7872.484527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7872.488486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7876.673155] LustreError: 429950:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 7876.765476] Lustre: server umount lustre-OST0000 complete [ 7881.185949] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7881.189638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7881.197945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7881.200950] Lustre: Skipped 1 previous similar message [ 7884.936704] LustreError: 430151:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7884.939622] LustreError: 430151:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7885.001259] Lustre: server umount lustre-MDT0000 complete [ 7887.061995] LustreError: 427679:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758744879 with bad export cookie 17612537625762854017 [ 7887.065113] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7887.068166] LustreError: 427679:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7887.152220] LustreError: 430353:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7887.156188] LustreError: 430353:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7887.268711] Lustre: server umount lustre-MDT0001 complete [ 7892.304576] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 7893.964110] Key type lgssc unregistered [ 7894.129043] LNet: 430885:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7894.132857] LNetError: 430885:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7895.209878] LNet: Removed LNI 192.168.204.140@tcp [ 7895.679184] Key type .llcrypt unregistered [ 7895.682830] Key type ._llcrypt unregistered [ 7907.597963] Key type ._llcrypt registered [ 7907.599544] Key type .llcrypt registered [ 7907.661252] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7908.160776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7908.259774] alg: No test for adler32 (adler32-zlib) [ 7909.193055] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 7909.359266] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 7910.975221] Key type lgssc registered [ 7911.562149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7916.058693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7916.079490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7917.386351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7919.581760] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7922.660252] LustreError: 432262:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7922.676757] LustreError: 432262:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 7924.334532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7924.548334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7926.686113] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7929.379389] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7931.170370] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7934.494447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7934.674250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7937.675781] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 7940.428908] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7941.865315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7949.988338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7949.992969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7950.007402] Lustre: Skipped 1 previous similar message [ 7953.000947] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.40@tcp (stopping) [ 7953.005805] Lustre: Skipped 2 previous similar messages [ 7955.140182] LustreError: 434508:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7955.206411] Lustre: server umount lustre-MDT0000 complete [ 7955.426883] LustreError: 432263:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7955.436511] LustreError: 432263:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 7956.567493] LustreError: 432244:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.40@tcp arrived at 1758744948 with bad export cookie 14413069519136867920 [ 7956.573305] LustreError: 432244:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7957.728142] LustreError: 432244:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758744950 with bad export cookie 14413069519136867276 [ 7957.735039] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7960.544580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7960.547683] LustreError: 432263:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7960.556106] Lustre: Skipped 1 previous similar message [ 7960.562136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7960.577360] LustreError: 432263:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 7962.596408] LustreError: 432263:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7962.597113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7962.604739] LustreError: 432263:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 7963.823931] LustreError: 434710:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7963.831846] LustreError: 434710:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7963.952290] Lustre: server umount lustre-OST0000 complete [ 7968.479228] LustreError: 432976:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000c7ab2ad x1844177693338880/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7968.499302] LustreError: 435064:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7968.506167] LustreError: 435064:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7968.629436] Lustre: server umount lustre-MDT0001 complete [ 7974.538704] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 7976.348245] Key type lgssc unregistered [ 7976.506246] LNet: 435596:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7976.511160] LNetError: 435596:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7976.521030] LNet: Removed LNI 192.168.204.140@tcp [ 7976.996786] Key type .llcrypt unregistered [ 7976.998840] Key type ._llcrypt unregistered [ 7989.716771] Key type ._llcrypt registered [ 7989.718199] Key type .llcrypt registered [ 7989.763844] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 7990.150846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7990.209865] alg: No test for adler32 (adler32-zlib) [ 7991.135202] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 7991.278387] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 7992.879185] Key type lgssc registered [ 7993.383761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7996.906255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7996.917264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7998.122219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7999.780926] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8003.506589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8003.680472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8005.305828] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8007.881508] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8009.527084] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8012.495047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8012.691679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8015.132374] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8017.723751] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8022.954883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8028.129608] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8028.134480] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8028.138993] Lustre: Skipped 1 previous similar message [ 8029.284369] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.40@tcp (stopping) [ 8029.288305] Lustre: Skipped 1 previous similar message [ 8033.257929] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8033.266847] Lustre: Skipped 1 previous similar message [ 8038.369258] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8038.374787] Lustre: Skipped 2 previous similar messages [ 8039.391359] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8039.458408] LustreError: 439216:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8039.514397] LustreError: 436965:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8039.518199] Lustre: server umount lustre-MDT0001 complete [ 8039.527290] LustreError: 436965:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 8043.488576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8043.488676] LustreError: 436970:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8043.488917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8043.491193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8048.609530] LustreError: 436970:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8048.645170] LustreError: 439422:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8048.652878] LustreError: 439422:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8048.757712] Lustre: server umount lustre-OST0000 complete [ 8050.602742] LustreError: 439623:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8050.605119] LustreError: 439623:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8050.735510] Lustre: server umount lustre-MDT0000 complete [ 8056.665281] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 8057.947246] Key type lgssc unregistered [ 8058.105382] LNet: 440306:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8058.109040] LNetError: 440306:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8058.116644] LNet: Removed LNI 192.168.204.140@tcp [ 8058.475115] Key type .llcrypt unregistered [ 8058.476646] Key type ._llcrypt unregistered [ 8069.377066] Key type ._llcrypt registered [ 8069.378630] Key type .llcrypt registered [ 8069.425816] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 8076.343590] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 8076.803301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8076.817793] alg: No test for adler32 (adler32-zlib) [ 8077.697933] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 8077.796671] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 8079.383225] Key type lgssc registered [ 8079.893575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8083.636366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8086.504332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8089.098104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8091.815390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8095.828472] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 8099.413173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8102.157066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8102.182874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8102.192592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8103.306140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8103.325437] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8103.366689] Lustre: lustre-MDT0000: new disk, initializing [ 8103.403664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8103.412882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8105.186779] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8109.669454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8112.788855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8112.826715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8112.958771] Lustre: lustre-OST0000: new disk, initializing [ 8112.961677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8112.964644] Lustre: Skipped 1 previous similar message [ 8112.996273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8115.355806] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8118.806943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8118.810939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8118.827916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8118.911453] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8122.727763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8125.394269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8125.424831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8125.467602] Lustre: 444252:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8125.494569] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8125.534578] Lustre: lustre-MDT0001: new disk, initializing [ 8125.583319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8125.598348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8125.601719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8127.255254] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8131.608986] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8131.696439] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8133.287977] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8133.389833] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8136.673517] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8136.673650] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8136.678021] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8136.682983] Lustre: Skipped 1 previous similar message [ 8136.687403] Lustre: Skipped 1 previous similar message [ 8139.877962] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.40@tcp (stopping) [ 8141.438215] LustreError: 447672:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8141.471892] Lustre: server umount lustre-MDT0001 complete [ 8146.913228] LustreError: 445267:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8146.921684] LustreError: 445267:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 8152.034039] LustreError: 444260:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8152.040579] LustreError: 444260:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 8154.591937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8154.595106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8154.600284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8157.155603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8157.156445] LustreError: 444265:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8160.300076] LustreError: 447945:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 8160.302929] LustreError: 447945:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8160.385748] Lustre: server umount lustre-OST0000 complete [ 8168.069752] LustreError: 448146:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 8168.071970] LustreError: 448146:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8168.189060] Lustre: server umount lustre-MDT0000 complete [ 8174.197833] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 8175.565398] Key type lgssc unregistered [ 8175.725668] LNet: 448831:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8175.730361] LNetError: 448831:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8175.745544] LNet: Removed LNI 192.168.204.140@tcp [ 8176.165780] Key type .llcrypt unregistered [ 8176.167309] Key type ._llcrypt unregistered [ 8192.968303] Key type ._llcrypt registered [ 8192.969804] Key type .llcrypt registered [ 8193.012601] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 8193.140910] Key type .llcrypt unregistered [ 8193.142593] Key type ._llcrypt unregistered [ 8200.856983] Key type ._llcrypt registered [ 8200.858492] Key type .llcrypt registered [ 8200.904338] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 8201.299807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8201.336117] alg: No test for adler32 (adler32-zlib) [ 8202.210462] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 8202.308219] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 8203.911180] Key type lgssc registered [ 8204.389827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8207.749580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8210.298426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8212.604048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8215.258576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8215.280957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8215.288352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8216.380427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8216.394524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8216.431789] Lustre: lustre-MDT0000: new disk, initializing [ 8216.459855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8216.469638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8217.966411] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8222.634218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8222.658519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8222.699974] Lustre: 452149:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8222.728727] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8222.731366] Lustre: Skipped 1 previous similar message [ 8222.767343] Lustre: lustre-MDT0001: new disk, initializing [ 8222.808269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8222.825724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8222.830165] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8224.399432] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8227.897617] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8229.250061] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8231.964694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8231.993239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8232.116707] Lustre: lustre-OST0000: new disk, initializing [ 8232.119525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8232.162906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8234.375454] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8237.891964] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8238.077884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8238.083261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8238.121336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8248.289448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8248.289495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8248.291295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8248.301895] Lustre: Skipped 1 previous similar message [ 8250.685460] LustreError: 454593:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8250.714820] Lustre: server umount lustre-MDT0000 complete [ 8251.986079] LustreError: 453896:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758745244 with bad export cookie 15478955072471897535 [ 8251.988611] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8251.991187] LustreError: 453896:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8252.057745] LustreError: 454794:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8252.059643] LustreError: 454794:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8252.133452] Lustre: server umount lustre-MDT0001 complete [ 8262.708124] LustreError: 454994:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8262.710902] LustreError: 454994:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8262.813381] Lustre: server umount lustre-OST0000 complete [ 8266.673243] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 16:20:58 (1758745258) [ 8267.059431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8269.451735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8269.663657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8271.048802] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8274.134825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8274.269477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8275.637165] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8277.407298] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8278.694711] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8281.239396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8281.275881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8281.359701] Lustre: Found index 0 for lustre-OST0000, updating log [ 8283.694117] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8286.001686] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8294.766549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8300.002849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8300.003104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8300.004327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8300.004332] Lustre: Skipped 2 previous similar messages [ 8300.005330] LustreError: Skipped 1 previous similar message [ 8300.015943] Lustre: Skipped 2 previous similar messages [ 8301.024505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8301.027635] Lustre: Skipped 2 previous similar messages [ 8304.109065] LustreError: 458098:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8304.111243] LustreError: 458098:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8304.190136] Lustre: server umount lustre-OST0000 complete [ 8305.325990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8307.235519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8307.236854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8307.241565] Lustre: Skipped 1 previous similar message [ 8312.288657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8312.291057] Lustre: Skipped 2 previous similar messages [ 8312.322624] LustreError: 458353:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8312.325565] LustreError: 458353:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8312.360185] Lustre: server umount lustre-MDT0000 complete [ 8313.707237] LustreError: 458355:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758745306 with bad export cookie 15478955072471898928 [ 8313.709632] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8313.711711] LustreError: 458355:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8313.874289] Lustre: server umount lustre-MDT0001 complete [ 8317.104963] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 16:21:48 (1758745308) [ 8317.457097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8317.809180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8318.163651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8318.507562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8318.861805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8319.225630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8319.602179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8320.001560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8320.400787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8320.787333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8321.184356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8321.546658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8321.913279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8322.284254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8322.635026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8322.996679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8323.365725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8323.716863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8324.076776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8324.457673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8324.831932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8325.218353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8325.619800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8325.981485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8326.344283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8326.701588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8327.054942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8327.407612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8327.769708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8328.132825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8328.518693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8328.890466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8329.262277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8329.653135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8330.011948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8330.365088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8330.755393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8331.150566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8331.499109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8331.836619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8332.195962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8332.599581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8333.030322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8333.464667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8333.890259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8334.327947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8334.765965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8335.178981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8335.563705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8335.919934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8336.308811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8339.141046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8339.355316] LustreError: 461938:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8339.362958] LustreError: 461938:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 8339.389953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8339.392689] Lustre: Skipped 1 previous similar message [ 8340.871055] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8344.018239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8345.605818] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8347.504164] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8348.861353] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8351.430634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8351.462401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8351.556945] Lustre: Found index 0 for lustre-OST0000, updating log [ 8353.748414] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8355.845587] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8358.885832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8358.889162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8358.900500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8362.673077] LustreError: 464246:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8362.679048] LustreError: 464246:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8362.764705] Lustre: server umount lustre-OST0000 complete [ 8365.667185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8365.674181] Lustre: Skipped 1 previous similar message [ 8370.784935] LustreError: 461937:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8370.793375] LustreError: 461937:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 8370.811573] Lustre: server umount lustre-MDT0000 complete [ 8371.808849] LustreError: 461937:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8371.814858] LustreError: 461937:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 8372.177326] LustreError: 463606:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758745364 with bad export cookie 15478955072471899999 [ 8372.180164] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8372.182091] LustreError: 463606:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8372.333146] Lustre: server umount lustre-MDT0001 complete [ 8373.355333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8378.902489] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 8382.695413] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 8386.772349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8389.206583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8391.547347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8393.969573] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8396.677137] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 16:23:08 (1758745388) [ 8420.590808] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 8421.816489] Key type lgssc unregistered [ 8421.957874] LNet: 468232:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8421.961202] LNetError: 468232:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8545.894923] LNet: Removed LNI 192.168.204.140@tcp [ 8546.226202] Key type .llcrypt unregistered [ 8546.227883] Key type ._llcrypt unregistered [ 8553.044480] Key type ._llcrypt registered [ 8553.045992] Key type .llcrypt registered [ 8553.088289] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 8553.489487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8553.517694] alg: No test for adler32 (adler32-zlib) [ 8554.403880] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 8554.491351] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 8556.079167] Key type lgssc registered [ 8556.474655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8558.079198] LNet: Added LNI 192.168.204.140@tcp100 [8/256/0/180] [ 8558.719348] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [ 8565.543057] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 8566.781066] Key type lgssc unregistered [ 8566.959108] LNet: 470098:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8566.963586] LNetError: 470098:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8566.966793] LNet: Removed LNI 192.168.204.140@tcp [ 8566.974527] LNet: Removed LNI 192.168.204.140@tcp100 [ 8567.267510] Key type .llcrypt unregistered [ 8567.268814] Key type ._llcrypt unregistered [ 8575.131704] Key type ._llcrypt registered [ 8575.132957] Key type .llcrypt registered [ 8575.171341] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 8581.397491] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 8581.840695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8581.870194] alg: No test for adler32 (adler32-zlib) [ 8582.740038] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 8582.834572] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 8584.415196] Key type lgssc registered [ 8584.854045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8588.176101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8590.537422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8592.656232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8594.998054] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8597.405289] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 16:26:29 (1758745589) [ 8603.769982] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 8604.884543] Key type lgssc unregistered [ 8605.021353] LNet: 474108:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8605.023521] LNetError: 474108:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8605.029373] LNet: Removed LNI 192.168.204.140@tcp [ 8605.343456] Key type .llcrypt unregistered [ 8605.344664] Key type ._llcrypt unregistered [ 8612.132677] Key type ._llcrypt registered [ 8612.133619] Key type .llcrypt registered [ 8612.171702] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 8612.622594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8612.664279] alg: No test for adler32 (adler32-zlib) [ 8613.523250] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 8613.615542] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 8615.199218] Key type lgssc registered [ 8615.631716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8618.754924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8618.773022] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8618.778825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8619.860460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8619.871671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8619.908372] Lustre: lustre-MDT0000: new disk, initializing [ 8619.935678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8619.944650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8621.318954] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8625.517197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8625.537744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8625.573679] Lustre: 475279:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8625.595814] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8625.598393] Lustre: Skipped 1 previous similar message [ 8625.629827] Lustre: lustre-MDT0001: new disk, initializing [ 8625.661335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8625.674039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8625.677729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8627.058931] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8630.233713] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8631.438471] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8633.720722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8633.752955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8633.851669] Lustre: lustre-OST0000: new disk, initializing [ 8633.854228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8633.887825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8635.822912] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8639.051451] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8639.493560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8639.498068] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8639.546925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8640.381880] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing lnet_if_list [ 8642.299213] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8642.312559] LNet: Added LNI 192.168.204.140@tcp42 [8/256/0/180] [ 8649.699137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8649.699312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8649.700028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8649.711171] Lustre: Skipped 1 previous similar message [ 8654.818501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8654.820659] Lustre: Skipped 2 previous similar messages [ 8655.022137] LustreError: 478660:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8655.093854] Lustre: server umount lustre-OST0000 complete [ 8659.938505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8659.939375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8659.944031] Lustre: Skipped 1 previous similar message [ 8659.948293] Lustre: Skipped 1 previous similar message [ 8662.653178] LustreError: 478866:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8662.655958] LustreError: 478866:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8662.691170] Lustre: server umount lustre-MDT0000 complete [ 8664.054395] LustreError: 475272:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758745656 with bad export cookie 2386038188411932163 [ 8664.057669] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8664.059230] LustreError: 475272:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8664.129845] LustreError: 479067:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8664.132897] LustreError: 479067:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8664.232543] Lustre: server umount lustre-MDT0001 complete [ 8668.195705] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 8669.365846] Key type lgssc unregistered [ 8669.547985] LNet: 479605:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8669.551057] LNetError: 479605:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8669.553669] LNet: Removed LNI 192.168.204.140@tcp [ 8669.559337] LNet: Removed LNI 192.168.204.140@tcp42 [ 8669.878809] Key type .llcrypt unregistered [ 8669.880504] Key type ._llcrypt unregistered [ 8677.855950] Key type ._llcrypt registered [ 8677.856929] Key type .llcrypt registered [ 8677.909730] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 8678.263089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8678.323760] alg: No test for adler32 (adler32-zlib) [ 8679.205223] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 8679.299634] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 8680.895090] Key type lgssc registered [ 8681.316137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8684.374734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8684.383996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8685.577272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8686.995968] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8690.182118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8690.331995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8691.771295] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8693.720970] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8694.996247] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8697.395815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8697.523380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8699.406246] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8701.198665] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8715.276846] loop0: detected capacity change from 0 to 204800000 [ 8715.332048] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8715.335993] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8715.340483] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8715.344775] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8715.365595] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8715.386426] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8715.399895] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8715.423702] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8715.448903] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8715.479811] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8717.367173] loop0: detected capacity change from 0 to 204800000 [ 8717.428520] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8717.486520] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8719.162737] loop0: detected capacity change from 0 to 204800000 [ 8720.954934] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8720.973195] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8721.009152] Lustre: fs15246-OST0000: new disk, initializing [ 8721.010875] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8721.024370] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8722.902932] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8726.711405] loop1: detected capacity change from 0 to 204800000 [ 8728.714836] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8728.739434] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8728.770280] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8728.791483] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8728.828411] Lustre: fs15246-MDT0000: new disk, initializing [ 8728.878908] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8728.888594] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8728.921370] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8729.951726] Lustre: 485223:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@00000000612b000c x1844178500745600/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1758745738 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8730.190115] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8739.039258] LustreError: 485223:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 8739.043098] LustreError: 485223:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 8739.045954] LustreError: 485223:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 8741.861356] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8741.866658] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8743.671750] LustreError: 486346:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 8743.697762] Lustre: server umount fs15246-MDT0000 complete [ 8746.808414] LustreError: 486846:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 8746.811078] LustreError: 486846:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8746.825401] Lustre: server umount fs15246-OST0000 complete [ 8751.071812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8751.074201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8751.077976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8752.101420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8752.101500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8752.104418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8756.334114] LustreError: 487348:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8756.336765] LustreError: 487348:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 8756.409765] Lustre: server umount lustre-OST0000 complete [ 8762.336514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8762.336564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8762.337416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8762.337421] Lustre: Skipped 1 previous similar message [ 8762.345702] Lustre: Skipped 1 previous similar message [ 8763.705924] LustreError: 487551:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8763.708949] LustreError: 487551:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8763.731620] Lustre: server umount lustre-MDT0000 complete [ 8764.919183] LustreError: 480805:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758745757 with bad export cookie 687370376213785986 [ 8764.922527] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8764.923804] LustreError: 480805:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8765.056376] Lustre: server umount lustre-MDT0001 complete [ 8768.599045] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 8769.697309] Key type lgssc unregistered [ 8769.833371] LNet: 488284:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8769.836062] LNetError: 488284:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8769.845363] LNet: Removed LNI 192.168.204.140@tcp [ 8770.180823] Key type .llcrypt unregistered [ 8770.182044] Key type ._llcrypt unregistered [ 8777.370847] Key type ._llcrypt registered [ 8777.371974] Key type .llcrypt registered [ 8777.406429] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 8777.703813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8777.730090] alg: No test for adler32 (adler32-zlib) [ 8778.576751] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 8778.650812] LNet: Added LNI 192.168.204.140@tcp [8/256/0/180] [ 8780.231130] Key type lgssc registered [ 8780.565440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8782.830476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8784.324154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8786.059378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8787.528546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8792.395087] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 8795.144356] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 8798.392222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8800.640312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8802.523295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8804.495871] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8806.727703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8806.746445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8806.752410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8807.835510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8807.846179] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8807.878673] Lustre: lustre-MDT0000: new disk, initializing [ 8807.901511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8807.909054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8809.150296] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8813.285232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8813.315370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8813.354960] Lustre: 492685:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8813.381317] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8813.383353] Lustre: Skipped 1 previous similar message [ 8813.419317] Lustre: lustre-MDT0001: new disk, initializing [ 8813.454546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8813.468445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8813.471920] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8814.767193] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8817.749457] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8818.921213] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8821.094097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8821.115149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8821.206568] Lustre: lustre-OST0000: new disk, initializing [ 8821.209179] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8821.238957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8821.943138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8821.945681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8821.987922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8823.104493] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_default_debug -1 all [ 8826.100965] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8827.441896] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8827.521764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8828.840850] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8828.916767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8832.481925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8832.481972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8832.486385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8832.488764] LustreError: Skipped 1 previous similar message [ 8835.629083] LustreError: 495621:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8835.718347] Lustre: server umount lustre-OST0000 complete [ 8837.600751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8837.601330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8837.605242] Lustre: Skipped 2 previous similar messages [ 8837.607621] Lustre: Skipped 1 previous similar message [ 8842.720176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8842.723162] Lustre: Skipped 1 previous similar message [ 8843.067135] LustreError: 495823:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8843.069092] LustreError: 495823:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8843.095351] Lustre: server umount lustre-MDT0000 complete [ 8844.305311] LustreError: 492676:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758745836 with bad export cookie 9895700839656516398 [ 8844.308874] LustreError: MGC192.168.204.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8844.309695] LustreError: 492676:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8844.373701] LustreError: 496025:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8844.375672] LustreError: 496025:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8844.455651] Lustre: server umount lustre-MDT0001 complete [ 8852.549306] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing set_hostid [ 8855.737118] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing load_modules_local [ 8859.640342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8862.424410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8864.588127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8866.854398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8867.442747] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8618 sec ========== 16:30:59 (1758745859) [ 8867.993096] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:30:59 (1758745859) === [ 8869.149191] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:31:00 (1758745860) === [ 8881.998852] Lustre: DEBUG MARKER: oleg440-server.virtnet: executing unload_modules_local [ 8883.025609] Key type lgssc unregistered [ 8883.159370] LNet: 500875:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8883.162067] LNetError: 500875:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8883.168329] LNet: Removed LNI 192.168.204.140@tcp [ 8883.477746] Key type .llcrypt unregistered [ 8883.478842] Key type ._llcrypt unregistered