[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 687557154 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002388] x2apic enabled [ 0.003000] Switched APIC routing to physical x2apic. [ 0.003010] kvm-guest: setup PV IPIs [ 0.005834] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.006018] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.007007] pid_max: default: 32768 minimum: 301 [ 0.008124] LSM: Security Framework initializing [ 0.009000] Yama: becoming mindful. [ 0.009053] SELinux: Initializing. [ 0.010095] *** VALIDATE selinux *** [ 0.018706] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022565] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023111] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025045] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026093] *** VALIDATE tmpfs *** [ 0.028120] *** VALIDATE proc *** [ 0.029202] *** VALIDATE cgroup *** [ 0.030004] *** VALIDATE cgroup2 *** [ 0.032062] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033117] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035039] Spectre V2 : User space: Vulnerable [ 0.036008] Speculative Store Bypass: Vulnerable [ 0.040046] debug: unmapping init [mem 0xffffffff98a59000-0xffffffff98a60fff] [ 0.042135] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043574] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044014] ... version: 2 [ 0.045007] ... bit width: 48 [ 0.046007] ... generic registers: 4 [ 0.047007] ... value mask: 0000ffffffffffff [ 0.048007] ... max period: 00007fffffffffff [ 0.049012] ... fixed-purpose events: 3 [ 0.050008] ... event mask: 000000070000000f [ 0.051229] rcu: Hierarchical SRCU implementation. [ 0.053787] smp: Bringing up secondary CPUs ... [ 0.054447] x86: Booting SMP configuration: [ 0.055031] .... node #0, CPUs: #1 #2 #3 [ 0.068566] smp: Brought up 1 node, 4 CPUs [ 0.070007] smpboot: Max logical packages: 1 [ 0.071013] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.117018] node 0 deferred pages initialised in 44ms [ 0.132031] devtmpfs: initialized [ 0.133220] x86/mm: Memory block size: 128MB [ 0.139234] gcov: version magic: 0x41383552 [ 0.144157] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148067] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153538] pinctrl core: initialized pinctrl subsystem [ 0.160155] [ 0.160619] ************************************************************* [ 0.164007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167009] ** ** [ 0.169008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170000] ** ** [ 0.172007] ** This means that this kernel is built to expose internal ** [ 0.174007] ** IOMMU data structures, which may compromise security on ** [ 0.176013] ** your system. ** [ 0.179024] ** ** [ 0.182008] ** If you see this message and you are not debugging the ** [ 0.184008] ** kernel, report this immediately to your vendor! ** [ 0.187008] ** ** [ 0.191010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193006] ************************************************************* [ 0.205423] NET: Registered protocol family 16 [ 0.207405] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.211038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.220062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.230049] cpuidle: using governor menu [ 0.231000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.234783] PCI: Using configuration type 1 for base access [ 0.240135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.261084] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.262000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.267647] cryptd: max_cpu_qlen set to 1000 [ 0.269622] ACPI: Added _OSI(Module Device) [ 0.270000] ACPI: Added _OSI(Processor Device) [ 0.270000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.271549] ACPI: Added _OSI(Processor Aggregator Device) [ 0.272000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.278677] ACPI: Interpreter enabled [ 0.280227] ACPI: PM: (supports S0 S3 S4 S5) [ 0.281008] ACPI: Using IOAPIC for interrupt routing [ 0.283063] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.286317] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.300224] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.303023] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.306011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.311167] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.318300] acpiphp: Slot [2] registered [ 0.320128] acpiphp: Slot [3] registered [ 0.324078] acpiphp: Slot [4] registered [ 0.328641] acpiphp: Slot [5] registered [ 0.329000] acpiphp: Slot [6] registered [ 0.333075] acpiphp: Slot [7] registered [ 0.335068] acpiphp: Slot [8] registered [ 0.338013] acpiphp: Slot [9] registered [ 0.344241] acpiphp: Slot [10] registered [ 0.350093] acpiphp: Slot [11] registered [ 0.353072] acpiphp: Slot [12] registered [ 0.355139] acpiphp: Slot [13] registered [ 0.356176] acpiphp: Slot [14] registered [ 0.358260] acpiphp: Slot [15] registered [ 0.361207] acpiphp: Slot [16] registered [ 0.363069] acpiphp: Slot [17] registered [ 0.366092] acpiphp: Slot [18] registered [ 0.368064] acpiphp: Slot [19] registered [ 0.370391] acpiphp: Slot [20] registered [ 0.372114] acpiphp: Slot [21] registered [ 0.376084] acpiphp: Slot [22] registered [ 0.378079] acpiphp: Slot [23] registered [ 0.380058] acpiphp: Slot [24] registered [ 0.384072] acpiphp: Slot [25] registered [ 0.388086] acpiphp: Slot [26] registered [ 0.390069] acpiphp: Slot [27] registered [ 0.394078] acpiphp: Slot [28] registered [ 0.396172] acpiphp: Slot [29] registered [ 0.400305] acpiphp: Slot [30] registered [ 0.402114] acpiphp: Slot [31] registered [ 0.403039] PCI host bridge to bus 0000:00 [ 0.406753] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.409009] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.411009] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.413011] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.416010] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.420014] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.422148] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.425000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.429168] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.435009] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.440044] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.442010] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.444007] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.447015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.450265] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.451736] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.453025] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.456516] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.460008] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.471010] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.475010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.480000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.499012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.511013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.525012] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.533225] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.538009] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.543010] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.553012] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.563009] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.571010] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.576010] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.586026] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.594441] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.599009] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.603046] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.613014] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.623538] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.628008] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.630973] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.639992] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.658390] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.670012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.681011] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.697015] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.716341] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.718360] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.725265] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.727387] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.729196] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.751052] iommu: Default domain type: Passthrough [ 0.753248] SCSI subsystem initialized [ 0.755107] ACPI: bus type USB registered [ 0.756058] usbcore: registered new interface driver usbfs [ 0.764108] usbcore: registered new interface driver hub [ 0.766044] usbcore: registered new device driver usb [ 0.770128] pps_core: LinuxPPS API ver. 1 registered [ 0.772007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.779052] PTP clock support registered [ 0.784099] EDAC MC: Ver: 3.0.0 [ 0.785427] PCI: Using ACPI for IRQ routing [ 0.790801] NetLabel: Initializing [ 0.792006] NetLabel: domain hash size = 128 [ 0.793007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.795052] NetLabel: unlabeled traffic allowed by default [ 0.805051] vgaarb: loaded [ 0.806258] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.807000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.826809] clocksource: Switched to clocksource kvm-clock [ 1.090029] VFS: Disk quotas dquot_6.6.0 [ 1.095264] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.100510] *** VALIDATE ramfs *** [ 1.101522] *** VALIDATE hugetlbfs *** [ 1.102571] pnp: PnP ACPI init [ 1.115580] pnp: PnP ACPI: found 6 devices [ 1.148210] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.158625] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.163818] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.169358] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.175106] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.180146] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.187485] NET: Registered protocol family 2 [ 1.193727] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.205289] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.207549] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.225188] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.227576] TCP: Hash tables configured (established 65536 bind 65536) [ 1.232768] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.234923] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.238207] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.242460] NET: Registered protocol family 1 [ 1.247324] RPC: Registered named UNIX socket transport module. [ 1.253019] RPC: Registered udp transport module. [ 1.256831] RPC: Registered tcp transport module. [ 1.260823] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.265826] NET: Registered protocol family 44 [ 1.266952] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.268341] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.269659] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.275658] PCI: CLS 0 bytes, default 64 [ 1.277894] Unpacking initramfs... [ 4.613646] debug: unmapping init [mem 0xffff994d3cc54000-0xffff994d3ffbffff] [ 4.623509] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.625525] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.639472] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 5.808819] Initialise system trusted keyrings [ 5.810188] Key type blacklist registered [ 5.812348] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.820853] zbud: loaded [ 5.824070] *** VALIDATE nfs *** [ 5.825034] *** VALIDATE nfs4 *** [ 5.826303] pstore: using deflate compression [ 5.829232] Platform Keyring initialized [ 6.021341] NET: Registered protocol family 38 [ 6.022835] Key type asymmetric registered [ 6.023997] Asymmetric key parser 'x509' registered [ 6.025734] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.029226] io scheduler mq-deadline registered [ 6.030704] io scheduler kyber registered [ 6.033138] io scheduler bfq registered [ 6.034775] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.037234] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.040881] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.043648] ACPI: Power Button [PWRF] [ 6.189066] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.385916] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.872568] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.013719] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.535388] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.600326] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.751576] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.758891] Non-volatile memory driver v1.3 [ 7.763701] Linux agpgart interface v0.103 [ 7.815607] virtio_blk virtio1: [vda] 133248 512-byte logical blocks (68.2 MB/65.1 MiB) [ 7.930151] vda: detected capacity change from 0 to 68222976 [ 7.942863] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.945538] vdb: detected capacity change from 0 to 1073741824 [ 7.970615] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.976484] vdc: detected capacity change from 0 to 2621440000 [ 7.996167] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.003167] vdd: detected capacity change from 0 to 2621440000 [ 8.020682] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.023365] vde: detected capacity change from 0 to 4294967296 [ 8.044106] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.051116] vdf: detected capacity change from 0 to 4294967296 [ 8.059957] libphy: Fixed MDIO Bus: probed [ 8.085128] usbcore: registered new interface driver usbserial_generic [ 8.087542] usbserial: USB Serial support registered for generic [ 8.089742] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.103985] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.105534] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.107652] mousedev: PS/2 mouse device common for all mice [ 8.111866] rtc_cmos 00:05: RTC can wake from S4 [ 8.118278] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.126747] rtc_cmos 00:05: registered as rtc0 [ 8.128158] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.130392] intel_pstate: CPU model not supported [ 8.141316] hid: raw HID events driver (C) Jiri Kosina [ 8.141760] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.143267] usbcore: registered new interface driver usbhid [ 8.153748] usbhid: USB HID core driver [ 8.155236] drop_monitor: Initializing network drop monitor service [ 8.157540] Initializing XFRM netlink socket [ 8.159361] NET: Registered protocol family 10 [ 8.161648] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.166776] Segment Routing with IPv6 [ 8.167965] NET: Registered protocol family 17 [ 8.174784] mpls_gso: MPLS GSO support [ 8.243889] RAS: Correctable Errors collector initialized. [ 8.245551] AVX version of gcm_enc/dec engaged. [ 8.246907] AES CTR mode by8 optimization enabled [ 8.500708] sched_clock: Marking stable (8500688925, 0)->(9821546089, -1320857164) [ 8.586239] registered taskstats version 1 [ 8.587920] Loading compiled-in X.509 certificates [ 8.589464] zswap: loaded using pool lzo/zbud [ 8.626606] Key type big_key registered [ 8.647684] Key type encrypted registered [ 8.648923] ima: No TPM chip found, activating TPM-bypass! [ 8.650526] ima: Allocated hash algorithm: sha1 [ 8.655323] ima: No architecture policies found [ 8.658140] evm: Initialising EVM extended attributes: [ 8.661686] evm: security.selinux [ 8.664079] evm: security.ima [ 8.664960] evm: security.capability [ 8.668083] evm: HMAC attrs: 0x1 [ 8.672873] rtc_cmos 00:05: setting system clock to 2025-10-10 23:06:16 UTC (1760137576) [ 8.684612] debug: unmapping init [mem 0xffffffff99a03000-0xffffffff99bfffff] [ 8.691206] debug: unmapping init [mem 0xffffffff98782000-0xffffffff98a58fff] [ 8.703123] Write protecting the kernel read-only data: 28672k [ 8.711187] debug: unmapping init [mem 0xffffffff96e03000-0xffffffff96ffffff] [ 8.715732] debug: unmapping init [mem 0xffffffff97714000-0xffffffff977fffff] [ 8.776241] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.791732] systemd[1]: Detected virtualization kvm. [ 8.795670] systemd[1]: Detected architecture x86-64. [ 8.797432] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.837290] systemd[1]: No hostname configured. [ 8.848629] systemd[1]: Set hostname to . [ 8.850391] random: systemd: uninitialized urandom read (16 bytes read) [ 8.852452] systemd[1]: Initializing machine ID from random generator. [ 9.053140] random: systemd: uninitialized urandom read (16 bytes read) [ 9.057444] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 9.063854] random: systemd: uninitialized urandom read (16 bytes read) [ 9.066570] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 9.072823] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.145469] device-mapper: uevent: version 1.0.3 [ 10.153123] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.166819] virtio_net virtio0 ens2: renamed from eth0 [ 11.237465] random: fast init done [ 11.375948] scsi host0: ata_piix [ 11.443792] scsi host1: ata_piix [ 11.454474] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.460812] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.181307] random: crng init done [ 16.184303] random: 7 urandom warning(s) missed due to ratelimiting [ 16.183777] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.727900] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.683467] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.352363] SELinux: Disabled at runtime. [ 20.467069] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.474877] systemd[1]: Detected virtualization kvm. [ 20.476688] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.836380] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.840784] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.846236] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.865266] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.870967] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.938477] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.972400] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ 22.407394] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 22.862877] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.571071] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.757365] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.071589] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.140664] EDAC sbridge: Ver: 1.1.2 [ 27.389816] Key type dns_resolver registered [ 28.164976] NFS: Registering the id_resolver key type [ 28.169505] Key type id_resolver registered [ 28.175966] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Crash recovery kernel arming. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg345-server login: [ 51.752267] libcfs: loading out-of-tree module taints kernel. [ 51.759228] Key type ._llcrypt registered [ 51.760060] Key type .llcrypt registered [ 51.788438] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 58.662155] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 59.356995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 59.363709] alg: No test for adler32 (adler32-zlib) [ 60.329136] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 60.608944] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 62.208130] Key type lgssc registered [ 62.740730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 68.630331] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 69.474226] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 72.660573] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 74.993476] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.468696] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 82.192907] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 86.300970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.322738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 86.330925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 87.413614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 87.425833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 87.464329] Lustre: lustre-MDT0000: new disk, initializing [ 87.485191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 87.491568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 88.711707] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 93.723893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.754626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 93.775290] Lustre: 6506:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 93.784231] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 93.785610] Lustre: Skipped 1 previous similar message [ 93.818553] Lustre: lustre-MDT0001: new disk, initializing [ 93.835044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 93.842031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 93.844732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 95.129747] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 97.455687] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 100.594241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 100.615134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 100.687105] Lustre: lustre-OST0000: new disk, initializing [ 100.688548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 100.700893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 102.485836] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 106.988076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 106.991424] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 107.018172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 107.974886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.003129] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 108.042323] Lustre: lustre-OST0001: new disk, initializing [ 108.045212] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 108.064107] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 109.992521] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 115.682392] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 116.715680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 116.718698] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 116.730786] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 121.170630] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 128.090474] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing check_logdir /tmp/testlogs/ [ 129.524984] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing yml_node [ 131.158763] Lustre: DEBUG MARKER: Client: 2.16.58.105 [ 132.203306] Lustre: DEBUG MARKER: MDS: 2.16.58.105 [ 133.138248] Lustre: DEBUG MARKER: OSS: 2.16.58.105 [ 133.766117] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 10 19:08:20 EDT 2025 [ 140.539722] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 141.069348] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 155.617322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 155.621096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 155.626781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 157.665052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 157.665761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 157.671031] Lustre: Skipped 1 previous similar message [ 157.673638] Lustre: Skipped 1 previous similar message [ 162.784467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 162.787402] Lustre: Skipped 5 previous similar messages [ 167.392153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 167.431564] LustreError: 12209:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 167.462881] Lustre: server umount lustre-MDT0000 complete [ 168.811193] LustreError: 6497:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760137736 with bad export cookie 11259102236816769879 [ 168.811279] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 168.815148] LustreError: 6497:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 173.024691] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 173.024920] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 173.025177] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 173.032284] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 173.036313] Lustre: Skipped 2 previous similar messages [ 178.144525] LustreError: 9132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 178.144542] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 178.150274] LustreError: 9132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 178.152057] Lustre: Skipped 2 previous similar messages [ 179.168804] LustreError: 9132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 179.174324] LustreError: 9132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 183.264123] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 183.297186] LustreError: 12412:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 183.299382] LustreError: 12412:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 183.367102] Lustre: server umount lustre-MDT0001 complete [ 184.810091] LustreError: 12616:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 184.813043] LustreError: 12616:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 184.832608] Lustre: server umount lustre-OST0000 complete [ 186.282604] Lustre: server umount lustre-OST0001 complete [ 188.517489] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 191.746985] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 195.594459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 198.104898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 200.264146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 202.431076] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 204.718065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 204.742038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 204.827332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 204.840482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 204.872955] Lustre: lustre-MDT0000: new disk, initializing [ 204.899476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.907080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 206.255274] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 210.459805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 210.484922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 210.524528] Lustre: 14994:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 210.622050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 211.924450] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 214.953973] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 216.108821] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 218.239407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 218.263577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 218.353464] Lustre: lustre-OST0000: new disk, initializing [ 218.354871] Lustre: Skipped 1 previous similar message [ 218.356623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 218.358917] Lustre: Skipped 2 previous similar messages [ 218.393957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.395927] Lustre: Skipped 1 previous similar message [ 219.585568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 219.587959] Lustre: Skipped 1 previous similar message [ 219.590789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 219.629879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 220.320416] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 223.506840] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 224.891346] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 224.963229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 226.192744] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 226.268863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 229.858623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 229.859480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 229.861323] LustreError: Skipped 1 previous similar message [ 229.867437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 229.870251] Lustre: Skipped 3 previous similar messages [ 241.120088] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 241.142087] LustreError: 17931:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 241.143860] LustreError: 17931:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 241.206649] Lustre: server umount lustre-OST0000 complete [ 245.218186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 245.222321] Lustre: Skipped 2 previous similar messages [ 248.638365] LustreError: 18134:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 248.640021] LustreError: 18134:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 248.668832] Lustre: server umount lustre-MDT0000 complete [ 250.048561] LustreError: 17932:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760137817 with bad export cookie 11259102236816771398 [ 250.050590] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 250.054858] LustreError: 17932:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 250.218242] Lustre: server umount lustre-MDT0001 complete [ 251.746958] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 19:10:18 (1760137818) [ 252.653167] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 253.301340] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 19:10:20 (1760137820) [ 255.614066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 255.761357] LustreError: 19004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 255.785772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 257.139766] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 260.061038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 261.498619] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 263.364440] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 264.732304] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 267.158926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 269.347490] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 271.201543] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 289.761108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 289.763133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.766813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 289.768732] Lustre: Skipped 6 previous similar messages [ 291.812493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 295.534874] LustreError: 21270:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 295.537611] LustreError: 21270:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 295.606056] Lustre: server umount lustre-OST0000 complete [ 311.264146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 311.325232] Lustre: server umount lustre-MDT0000 complete [ 312.290140] LustreError: 19003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.299224] LustreError: 19003:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 312.643271] LustreError: 19030:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760137880 with bad export cookie 11259102236816772280 [ 312.646500] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 312.646620] LustreError: 19030:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 312.710194] LustreError: 21676:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 312.712477] LustreError: 21676:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 312.811792] Lustre: server umount lustre-MDT0001 complete [ 316.723956] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 317.840057] Key type lgssc unregistered [ 317.982442] LNet: 22207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.986421] LNetError: 22207:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 317.995383] LNet: Removed LNI 192.168.203.145@tcp [ 318.386119] Key type .llcrypt unregistered [ 318.387240] Key type ._llcrypt unregistered [ 327.186735] Key type ._llcrypt registered [ 327.187853] Key type .llcrypt registered [ 327.233616] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 327.690546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 327.732370] alg: No test for adler32 (adler32-zlib) [ 328.602427] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 328.698066] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 330.288178] Key type lgssc registered [ 330.696789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 333.511652] loop: module loaded [ 333.529993] loop0: detected capacity change from 0 to 204800000 [ 333.601464] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 333.606721] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 333.734250] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 333.738864] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 333.762795] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 333.782959] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 333.799234] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 333.822133] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 335.774048] loop0: detected capacity change from 0 to 204800000 [ 335.829353] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 335.833476] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 335.837232] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 335.888105] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 337.645067] loop0: detected capacity change from 0 to 204800000 [ 339.552184] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 339.574148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 339.579795] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 340.679277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 340.691601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 340.729135] Lustre: lustre-MDT0000: new disk, initializing [ 340.754675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 340.762676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 342.124282] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 345.713514] loop1: detected capacity change from 0 to 204800000 [ 347.679434] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 347.703923] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 347.810865] Lustre: lustre-OST07c6: new disk, initializing [ 347.814385] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 347.816500] Lustre: Skipped 1 previous similar message [ 347.839799] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 349.755713] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 354.817782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 354.821493] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 354.834807] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 439.980174] LustreError: 26536:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 440.044774] Lustre: server umount lustre-OST07c6 complete [ 452.579942] LustreError: 27036:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 452.581990] LustreError: 27036:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 452.662288] Lustre: server umount lustre-MDT0000 complete [ 457.776298] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 458.784411] Key type lgssc unregistered [ 458.912343] LNet: 27868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.914814] LNetError: 27868:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 458.921332] LNet: Removed LNI 192.168.203.145@tcp [ 459.225053] Key type .llcrypt unregistered [ 459.226029] Key type ._llcrypt unregistered [ 466.141451] Key type ._llcrypt registered [ 466.142509] Key type .llcrypt registered [ 466.176143] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 466.520871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 466.552420] alg: No test for adler32 (adler32-zlib) [ 467.404610] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 467.482552] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 469.056149] Key type lgssc registered [ 469.389502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 471.991397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 471.998304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.157289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 474.390782] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 477.060950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 477.172939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 478.386891] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 480.128421] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 481.209768] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 483.242796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 483.351274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.316124] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 487.227473] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 497.706481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 504.580326] LustreError: 31315:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 507.876165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 507.876239] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.877517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 507.887171] Lustre: Skipped 1 previous similar message [ 512.995190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 512.997383] Lustre: Skipped 2 previous similar messages [ 513.390986] LustreError: 31569:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 513.451534] Lustre: server umount lustre-OST0000 complete [ 518.115615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.115843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 518.122577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 518.125108] Lustre: Skipped 1 previous similar message [ 520.894406] LustreError: 31771:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 520.896452] LustreError: 31771:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 520.927052] Lustre: server umount lustre-MDT0000 complete [ 522.206335] LustreError: 29817:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138090 with bad export cookie 15866142112449290246 [ 522.209789] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 522.209887] LustreError: 29817:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 522.276866] LustreError: 31973:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 522.278917] LustreError: 31973:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 522.364566] Lustre: server umount lustre-MDT0001 complete [ 526.035583] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 527.297516] Key type lgssc unregistered [ 527.455404] LNet: 32503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 527.459173] LNetError: 32503:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 527.468335] LNet: Removed LNI 192.168.203.145@tcp [ 527.820151] Key type .llcrypt unregistered [ 527.821161] Key type ._llcrypt unregistered [ 535.259315] Key type ._llcrypt registered [ 535.260895] Key type .llcrypt registered [ 535.306120] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 535.830876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 535.848429] alg: No test for adler32 (adler32-zlib) [ 536.709301] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 536.796900] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 538.376152] Key type lgssc registered [ 538.767248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 541.484030] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 541.491369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.654030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 543.943166] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 546.760450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 546.887964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 548.195917] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 550.003256] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 551.235310] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 553.573134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 553.721439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.640651] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 557.530255] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 568.041680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 572.385500] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 572.389480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 572.392971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 573.411069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 573.411739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 573.414148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.997752] LustreError: 35999:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 574.057239] Lustre: server umount lustre-OST0000 complete [ 577.470433] LustreError: 36199:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 577.472907] LustreError: 36199:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 577.498944] Lustre: server umount lustre-MDT0000 complete [ 577.505482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 577.510283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.518034] LustreError: 34454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.525730] LustreError: 34454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 578.529061] LustreError: 34454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.533797] LustreError: 34454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 578.806362] LustreError: 33712:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138146 with bad export cookie 15336073287280971077 [ 578.810507] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.812279] LustreError: 33712:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 578.873876] LustreError: 36400:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 578.875803] LustreError: 36400:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 578.953911] Lustre: server umount lustre-MDT0001 complete [ 582.683135] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 583.786116] Key type lgssc unregistered [ 583.922393] LNet: 36930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 583.924560] LNetError: 36930:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 583.935345] LNet: Removed LNI 192.168.203.145@tcp [ 584.238830] Key type .llcrypt unregistered [ 584.240078] Key type ._llcrypt unregistered [ 593.337235] Key type ._llcrypt registered [ 593.338441] Key type .llcrypt registered [ 593.380376] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 593.670610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 593.681742] alg: No test for adler32 (adler32-zlib) [ 594.552303] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 594.643428] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 596.232147] Key type lgssc registered [ 596.645376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 599.282733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 599.290108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 600.458052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.778247] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 604.760487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 604.893355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 606.284920] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 608.113691] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 609.411790] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 611.698832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 611.814257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 613.752525] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 615.618123] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 627.176470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 635.872756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 635.875622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.880553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 637.409371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 637.411258] Lustre: Skipped 1 previous similar message [ 637.411308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.418959] Lustre: Skipped 1 previous similar message [ 639.548330] LustreError: 40573:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 639.576961] Lustre: server umount lustre-MDT0000 complete [ 640.828928] LustreError: 39924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138208 with bad export cookie 11944391321249038886 [ 640.832675] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 640.833387] LustreError: 39924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 641.450661] LustreError: 38299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 641.455815] LustreError: 38299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 641.459075] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 641.462939] Lustre: Skipped 1 previous similar message [ 642.529117] LustreError: 39668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.529396] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 645.536798] LustreError: 39668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 645.537047] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 645.543286] Lustre: Skipped 1 previous similar message [ 650.657297] LustreError: 40403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.657350] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 650.663682] LustreError: 40403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 650.666016] Lustre: Skipped 1 previous similar message [ 655.328191] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 655.365647] LustreError: 40776:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 655.369166] LustreError: 40776:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 655.459660] Lustre: server umount lustre-MDT0001 complete [ 667.223652] LustreError: 41001:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 667.226183] LustreError: 41001:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 667.289882] Lustre: server umount lustre-OST0000 complete [ 672.760594] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 673.928970] Key type lgssc unregistered [ 674.071370] LNet: 41837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.075342] LNetError: 41837:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 674.085333] LNet: Removed LNI 192.168.203.145@tcp [ 674.379789] Key type .llcrypt unregistered [ 674.381229] Key type ._llcrypt unregistered [ 682.975292] Key type ._llcrypt registered [ 682.976528] Key type .llcrypt registered [ 683.016404] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 683.430695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 683.461469] alg: No test for adler32 (adler32-zlib) [ 684.315023] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 684.398471] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 685.976167] Key type lgssc registered [ 686.350788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 689.046343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 689.053226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.214243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 691.531817] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 694.514024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 694.648309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 696.037123] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 697.923738] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 699.147788] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 701.404385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 701.519542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 703.529511] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 705.409663] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 715.875984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 725.473748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 725.476201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.479852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 726.497790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 726.499387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 726.500795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 726.511593] Lustre: Skipped 2 previous similar messages [ 729.132585] LustreError: 45491:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 729.201699] Lustre: server umount lustre-OST0000 complete [ 742.022384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 742.023788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 742.025637] Lustre: Skipped 1 previous similar message [ 747.489332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 747.491280] Lustre: Skipped 2 previous similar messages [ 748.220281] LustreError: 45866:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 748.222131] LustreError: 45866:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 748.249786] Lustre: server umount lustre-MDT0000 complete [ 749.512413] LustreError: 43202:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138317 with bad export cookie 12385074848223908455 [ 749.517203] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 755.647305] LustreError: 46067:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 755.649093] LustreError: 46067:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 755.739530] Lustre: server umount lustre-MDT0001 complete [ 759.637804] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 760.837936] Key type lgssc unregistered [ 760.994386] LNet: 46599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.996784] LNetError: 46599:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 761.005415] LNet: Removed LNI 192.168.203.145@tcp [ 761.347529] Key type .llcrypt unregistered [ 761.349151] Key type ._llcrypt unregistered [ 770.146926] Key type ._llcrypt registered [ 770.147957] Key type .llcrypt registered [ 770.185664] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 770.602818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 770.612587] alg: No test for adler32 (adler32-zlib) [ 771.479464] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 771.562803] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 773.144168] Key type lgssc registered [ 773.548039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 776.232755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 776.241197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 777.395113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 778.641265] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 781.490867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 781.610890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 782.912139] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 784.782360] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 786.000780] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 788.278422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 788.398292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 790.312308] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 792.040725] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 802.727220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 804.818432] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 805.967111] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 816.735255] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-1019h19m43s [ 817.265929] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 818.099234] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 818.146397] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 818.150339] Lustre: Skipped 2 previous similar messages [ 819.114142] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 819.117056] Lustre: Skipped 3 previous similar messages [ 824.187936] LustreError: 50560:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 824.213203] Lustre: server umount lustre-MDT0000 complete [ 825.257825] LustreError: 47997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 825.263430] LustreError: 47997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 825.325931] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 827.611550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 827.656149] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 827.746313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 827.778806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 828.775660] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 829.066987] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 830.189309] Lustre: DEBUG MARKER: Wait for df (35955) ... [ 830.692670] Lustre: DEBUG MARKER: done [ 832.480791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 832.483295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 832.487581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 832.489550] Lustre: Skipped 3 previous similar messages [ 832.993090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 837.677117] LustreError: 51608:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 837.680415] LustreError: 51608:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 837.743182] Lustre: server umount lustre-OST0000 complete [ 839.137558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 839.137660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 839.144607] Lustre: Skipped 1 previous similar message [ 839.146713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 839.148747] Lustre: Skipped 4 previous similar messages [ 844.193326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 844.195387] Lustre: Skipped 2 previous similar messages [ 845.052396] LustreError: 51813:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 845.054447] LustreError: 51813:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 845.080790] Lustre: server umount lustre-MDT0000 complete [ 845.217388] LustreError: 50357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 845.225098] LustreError: 50357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 846.315301] LustreError: 50290:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138414 with bad export cookie 18231202740980442349 [ 846.318807] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 846.318964] LustreError: 50290:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 846.468883] Lustre: server umount lustre-MDT0001 complete [ 850.177494] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 851.280352] Key type lgssc unregistered [ 851.411416] LNet: 52545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.413685] LNetError: 52545:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.419415] LNet: Removed LNI 192.168.203.145@tcp [ 851.733044] Key type .llcrypt unregistered [ 851.734226] Key type ._llcrypt unregistered [ 862.816519] Key type ._llcrypt registered [ 862.817963] Key type .llcrypt registered [ 862.858484] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 868.977601] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 869.286627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 869.317023] alg: No test for adler32 (adler32-zlib) [ 870.167886] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 870.250241] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 871.832196] Key type lgssc registered [ 872.187549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 875.227318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 877.547376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 879.717335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 882.007363] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 884.282366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 884.304686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 884.309712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 885.389702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 885.401225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 885.435710] Lustre: lustre-MDT0000: new disk, initializing [ 885.459651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 885.466668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 886.684737] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 890.661149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 890.684970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 890.717666] Lustre: 56468:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 890.739625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 890.742029] Lustre: Skipped 1 previous similar message [ 890.774242] Lustre: lustre-MDT0001: new disk, initializing [ 890.803152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 890.814141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 890.816964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 892.011025] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 894.961732] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 896.108432] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 898.296388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 898.319148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 898.406686] Lustre: lustre-OST0000: new disk, initializing [ 898.408498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 898.434031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 900.202209] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 903.119315] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 903.160956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 903.163765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 903.196477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 904.374546] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 904.437713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 905.682434] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 905.763164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 908.258968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 908.259485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 908.261722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 908.264273] Lustre: Skipped 1 previous similar message [ 912.431066] LustreError: 59407:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 912.490165] Lustre: server umount lustre-OST0000 complete [ 916.449239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 916.451862] LustreError: Skipped 1 previous similar message [ 916.453309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 916.458632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 916.460605] Lustre: Skipped 1 previous similar message [ 918.497174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 918.497663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.617608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 923.619816] Lustre: Skipped 2 previous similar messages [ 928.224071] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 928.260937] LustreError: 59608:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 928.264015] LustreError: 59608:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 928.290867] Lustre: server umount lustre-MDT0000 complete [ 929.549212] LustreError: 57248:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138497 with bad export cookie 3351330493112392059 [ 929.552439] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 929.552984] LustreError: 57248:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 929.615463] LustreError: 59811:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 929.617854] LustreError: 59811:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 929.703755] Lustre: server umount lustre-MDT0001 complete [ 932.646422] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 19:21:39 (1760138499) [ 934.968738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 935.133061] LustreError: 60480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 935.138505] LustreError: 60480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 935.156205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 936.511031] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 939.578761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 939.703328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 941.085436] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 942.941687] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 944.150071] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 946.409489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 948.440549] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 950.309176] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 963.321064] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-1019h22m10s [ 963.811318] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 966.129623] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 966.974411] Lustre: DEBUG MARKER: Creating a test file and stat it [ 966.988914] LustreError: 62635:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 987.060497] Lustre: lustre-MDT0000: Client b6a388a2-d5e6-4467-b3c1-1d34cce89b91 (at 192.168.203.45@tcp) reconnecting [ 1007.032153] LustreError: 62635:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1007.758235] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1008.602686] Lustre: DEBUG MARKER: done [ 1011.171194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1011.172243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1011.175512] LustreError: Skipped 1 previous similar message [ 1011.181751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1011.185857] Lustre: Skipped 1 previous similar message [ 1016.303984] LustreError: 63503:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1016.305739] LustreError: 63503:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1016.372277] Lustre: server umount lustre-OST0000 complete [ 1021.409610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1021.410421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1021.414533] Lustre: Skipped 2 previous similar messages [ 1021.416657] Lustre: Skipped 2 previous similar messages [ 1023.844919] LustreError: 63706:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1023.850455] LustreError: 63706:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1023.926778] Lustre: server umount lustre-MDT0000 complete [ 1026.577887] LustreError: 60460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138594 with bad export cookie 3351330493112392927 [ 1026.581171] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1026.600191] LustreError: 60460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1027.093954] Lustre: server umount lustre-MDT0001 complete [ 1035.149557] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1037.595067] Key type lgssc unregistered [ 1037.845494] LNet: 64441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.852019] LNetError: 64441:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.864374] LNet: Removed LNI 192.168.203.145@tcp [ 1038.510354] Key type .llcrypt unregistered [ 1038.511531] Key type ._llcrypt unregistered [ 1060.885068] Key type ._llcrypt registered [ 1060.886611] Key type .llcrypt registered [ 1060.942835] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 1067.953925] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1068.465155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1068.479062] alg: No test for adler32 (adler32-zlib) [ 1069.352208] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1069.447781] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1071.048158] Key type lgssc registered [ 1071.471945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1074.720788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1077.148814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1079.390309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1081.674534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1083.962424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1083.981028] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1083.988196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.081223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1085.095262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1085.136359] Lustre: lustre-MDT0000: new disk, initializing [ 1085.167371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1085.175939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1086.634707] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1090.856173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1090.880445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1090.914396] Lustre: 68363:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1090.947731] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1090.949807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1090.953133] Lustre: Skipped 1 previous similar message [ 1090.990016] Lustre: lustre-MDT0001: new disk, initializing [ 1091.026679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1091.039916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1091.043934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1092.564310] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1095.993515] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1097.326501] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1100.019929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1100.055829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1100.173757] Lustre: lustre-OST0000: new disk, initializing [ 1100.177306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1100.216342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1102.485277] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1106.049193] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1106.946429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1106.950576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1106.993897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1107.718207] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1107.805601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1109.349682] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1109.436131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1112.033654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1112.035520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1112.036694] LustreError: Skipped 1 previous similar message [ 1112.041988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1112.045269] Lustre: Skipped 1 previous similar message [ 1116.334067] LustreError: 71299:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1116.396484] Lustre: server umount lustre-OST0000 complete [ 1121.760749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1121.762860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.766951] Lustre: Skipped 1 previous similar message [ 1121.768904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1123.837807] LustreError: 71500:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1123.840282] LustreError: 71500:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1123.867363] Lustre: server umount lustre-MDT0000 complete [ 1125.158241] LustreError: 69144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138692 with bad export cookie 10354239868304578870 [ 1125.161680] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1125.162635] LustreError: 69144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1125.227412] LustreError: 71703:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1125.229949] LustreError: 71703:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1125.318928] Lustre: server umount lustre-MDT0001 complete [ 1128.007559] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 19:24:54 (1760138694) [ 1128.557900] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1129.152521] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 19:24:56 (1760138696) [ 1132.848270] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1134.225108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1134.374267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1134.725309] Lustre: Failing over lustre-MDT0000 [ 1134.727793] LustreError: 72777:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1134.732072] LustreError: 72777:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1134.734714] LustreError: 72777:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1134.794791] LustreError: 72828:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1134.797049] LustreError: 72828:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1134.873407] Lustre: server umount lustre-MDT0000 complete [ 1136.937170] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:25:03 (1760138703) [ 1139.107780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1139.238407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.432931] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1143.335172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1144.724582] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1146.490362] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1147.658068] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1149.930797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1150.042690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1150.044782] Lustre: Skipped 1 previous similar message [ 1151.995304] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1153.786239] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1171.396320] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1174.496921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1174.499753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1174.505302] Lustre: Skipped 1 previous similar message [ 1174.507463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1174.510274] Lustre: Skipped 2 previous similar messages [ 1179.261680] LustreError: 75767:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1179.264367] LustreError: 75767:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1179.292727] Lustre: server umount lustre-MDT0000 complete [ 1180.129845] LustreError: 73396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1180.135975] LustreError: 73396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1180.511989] LustreError: 73378:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138748 with bad export cookie 10354239868304580025 [ 1180.514954] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1180.515926] LustreError: 73378:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1180.651107] Lustre: server umount lustre-MDT0001 complete [ 1181.767323] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1183.408681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1186.137391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.368993] LustreError: 66577:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000077dd3f85 x1845639131491072/t0(0) o250->MGC192.168.203.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1190.488510] LustreError: 75002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1190.519603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1190.548641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1191.914555] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1194.821025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1194.916883] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1194.923132] Lustre: Skipped 2 previous similar messages [ 1196.314356] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1198.063973] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1199.227864] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1200.102544] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1202.144161] Lustre: 66579:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760138753/real 1760138753] req@00000000c6291587 x1845639131490816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760138769 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1205.223878] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1206.241256] Lustre: 66579:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760138758/real 1760138758] req@00000000c1da51c9 x1845639131490944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760138774 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1210.339947] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1210.341646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1210.342934] Lustre: Skipped 2 previous similar messages [ 1210.343570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1210.343591] Lustre: Skipped 3 previous similar messages [ 1212.424542] LustreError: 78226:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1212.428317] LustreError: 78226:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1212.463535] Lustre: server umount lustre-MDT0000 complete [ 1213.762288] LustreError: 73376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138781 with bad export cookie 10354239868304581915 [ 1213.765973] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1213.767071] LustreError: 73376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1213.910169] Lustre: server umount lustre-MDT0001 complete [ 1216.068746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1219.398252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.584572] LustreError: 66577:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007f647236 x1845639131510528/t0(0) o250->MGC192.168.203.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1223.700505] LustreError: 77544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1223.727114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1223.730325] Lustre: Skipped 1 previous similar message [ 1223.756324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1225.040388] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1227.895368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1227.982175] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.355159] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1231.080286] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1231.199564] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1232.096173] Lustre: 66578:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760138783/real 1760138783] req@00000000801fda4a x1845639131510144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760138799 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1232.459274] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1235.232163] Lustre: 66580:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760138786/real 1760138786] req@00000000c36d337e x1845639131510272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760138802 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1236.454227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1236.457369] Lustre: Skipped 4 previous similar messages [ 1239.520128] Lustre: 66579:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760138791/real 1760138791] req@00000000da94ed4c x1845639131510656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760138807 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1239.529956] Lustre: 66579:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1240.320852] LustreError: 80680:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1240.322911] LustreError: 80680:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1240.354821] Lustre: server umount lustre-MDT0000 complete [ 1241.571657] LustreError: 75001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.574092] LustreError: 75972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138809 with bad export cookie 10354239868304583287 [ 1241.576618] LustreError: 75001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1241.577814] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1241.581533] LustreError: 75972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1241.706370] Lustre: server umount lustre-MDT0001 complete [ 1243.873180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1244.992211] Lustre: DEBUG MARKER: files compared the same [ 1255.987993] Lustre: server umount lustre-OST0000 complete [ 1261.529947] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1262.536642] Key type lgssc unregistered [ 1262.664332] LNet: 82310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1262.666587] LNetError: 82310:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1262.677413] LNet: Removed LNI 192.168.203.145@tcp [ 1262.981321] Key type .llcrypt unregistered [ 1262.983236] Key type ._llcrypt unregistered [ 1271.405994] Key type ._llcrypt registered [ 1271.407742] Key type .llcrypt registered [ 1271.445982] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1271.791907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1271.840306] alg: No test for adler32 (adler32-zlib) [ 1272.698202] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1272.775027] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1274.352215] Key type lgssc registered [ 1274.697547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1277.270305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1277.276816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1278.424087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1279.729649] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug +malloc all [ 1282.665706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1282.763474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1284.151634] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug +malloc all [ 1286.031721] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1287.333645] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1289.692171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1289.787304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1291.612673] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug +malloc all [ 1293.352632] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1305.125093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1313.248567] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1313.251541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.256321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1315.297142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1315.297494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1315.297561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1315.299649] Lustre: Skipped 1 previous similar message [ 1317.612968] LustreError: 85951:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1317.665892] Lustre: server umount lustre-OST0000 complete [ 1320.416816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1320.417119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1320.420975] Lustre: Skipped 1 previous similar message [ 1320.424486] Lustre: Skipped 1 previous similar message [ 1325.536526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1325.539920] Lustre: Skipped 1 previous similar message [ 1330.656597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1330.659637] Lustre: Skipped 1 previous similar message [ 1333.216142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1333.253080] LustreError: 86154:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1333.255575] LustreError: 86154:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1333.282790] Lustre: server umount lustre-MDT0000 complete [ 1334.536669] LustreError: 83665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138902 with bad export cookie 11084320573199870245 [ 1334.538149] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1334.540735] LustreError: 83665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1334.602379] LustreError: 86357:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1334.605636] LustreError: 86357:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1334.676598] Lustre: server umount lustre-MDT0001 complete [ 1338.436367] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1339.616434] Key type lgssc unregistered [ 1339.761321] LNet: 86889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1339.764120] LNetError: 86889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1339.771293] LNet: Removed LNI 192.168.203.145@tcp [ 1340.089559] Key type .llcrypt unregistered [ 1340.091242] Key type ._llcrypt unregistered [ 1349.233288] Key type ._llcrypt registered [ 1349.234713] Key type .llcrypt registered [ 1349.280824] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1349.767535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1349.780544] alg: No test for adler32 (adler32-zlib) [ 1350.658375] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1350.742815] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1352.328121] Key type lgssc registered [ 1352.688199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1355.221546] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1355.228059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1370.656394] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1375.776307] LustreError: 88236:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [ 1402.400200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.441889] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1406.311574] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1408.886092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1408.925571] Lustre: *** cfs_fail_loc=706, val=0*** [ 1409.066960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1409.087912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1410.406602] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1413.296298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1413.461494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1414.698314] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1416.427899] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1417.641224] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1418.722729] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1418.727272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1418.735234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1419.232894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1419.235570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.241468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1420.527126] LustreError: 90557:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1420.587885] Lustre: server umount lustre-OST0000 complete [ 1423.841424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1423.846062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1423.852606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1426.913253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1426.913693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.033357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1432.035372] Lustre: Skipped 2 previous similar messages [ 1436.128146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1436.162074] LustreError: 90759:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1436.163905] LustreError: 90759:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1436.189205] Lustre: server umount lustre-MDT0000 complete [ 1437.153553] LustreError: 90761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1437.159608] LustreError: 90761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1437.427550] LustreError: 88244:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139005 with bad export cookie 5784375404944355220 [ 1437.431025] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1437.433969] LustreError: 88244:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1437.497693] LustreError: 90962:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1437.499819] LustreError: 90962:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1437.596480] Lustre: server umount lustre-MDT0001 complete [ 1441.206438] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1442.348446] Key type lgssc unregistered [ 1442.480334] LNet: 91494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1442.482811] LNetError: 91494:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1442.489296] LNet: Removed LNI 192.168.203.145@tcp [ 1442.793178] Key type .llcrypt unregistered [ 1442.795173] Key type ._llcrypt unregistered [ 1451.894630] Key type ._llcrypt registered [ 1451.895694] Key type .llcrypt registered [ 1451.937867] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1452.461628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1452.492629] alg: No test for adler32 (adler32-zlib) [ 1453.377815] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1453.473605] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1455.056142] Key type lgssc registered [ 1455.512942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1458.665246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1458.672246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1460.998379] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1463.507059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1463.700579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1464.997423] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1467.821617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1467.937577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1469.749069] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1472.474764] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1472.478964] Lustre: MGS: Client 03c056ab-c3f0-4028-a88b-f9126cea1714 (at 0@lo) reconnecting [ 1472.565539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.595968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1473.926574] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1482.722333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1482.725291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1482.730181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1483.750636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1483.751375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1483.751920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1487.728121] LustreError: 95506:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1487.808455] Lustre: server umount lustre-OST0000 complete [ 1493.986096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1493.986333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1493.987249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1493.987256] Lustre: Skipped 1 previous similar message [ 1493.996719] Lustre: Skipped 1 previous similar message [ 1495.236056] LustreError: 95707:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1495.238975] LustreError: 95707:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1495.248978] Lustre: server umount lustre-MDT0000 complete [ 1496.563488] LustreError: 95910:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1496.565368] LustreError: 95910:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1496.636076] Lustre: server umount lustre-MDT0001 complete [ 1498.178765] Lustre: server umount MGS complete [ 1502.222851] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 19:31:09 (1760139069) [ 1513.206195] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 1517.193795] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1520.513604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1522.959953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1525.090333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1527.381588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1529.860091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1529.890944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1531.167547] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1535.089761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1535.115192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1535.212564] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1535.229452] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1535.268952] Lustre: lustre-MDT0001: new disk, initializing [ 1535.296837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1535.306577] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1535.308488] Lustre: 100618:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1535.310711] LustreError: 100618:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1536.715527] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1541.653250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1541.690269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1541.812540] Lustre: lustre-OST0000: new disk, initializing [ 1541.814586] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1541.850799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1544.333310] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1547.808788] Lustre: 101876:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@00000000f8117602 x1845639534118016/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1760139131 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1547.951289] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1550.179217] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1550.183270] Lustre: MGS: Client 233348d4-1700-4318-9e65-92d92c8372a7 (at 0@lo) reconnecting [ 1550.195890] Lustre: 99903:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1550.203268] LustreError: 99903:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1550.213225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1550.249796] Lustre: lustre-MDT0000: new disk, initializing [ 1550.296820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1550.305611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1551.329832] Lustre: 102334:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000004be41878 x1845639534123264/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1760139135 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1551.392464] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1551.572948] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1556.388312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1556.395114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1556.459175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1576.929287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1576.933509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.941845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1576.946352] Lustre: Skipped 1 previous similar message [ 1581.564138] LustreError: 102854:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1581.568629] LustreError: 102854:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1581.655857] Lustre: server umount lustre-OST0000 complete [ 1585.121210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1585.123855] LustreError: Skipped 1 previous similar message [ 1585.125455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1585.129863] Lustre: Skipped 1 previous similar message [ 1585.131893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1585.134898] Lustre: Skipped 3 previous similar messages [ 1589.316745] LustreError: 103056:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1589.319643] LustreError: 103056:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1589.334619] Lustre: server umount lustre-MDT0000 complete [ 1596.879448] Lustre: server umount lustre-MDT0001 complete [ 1598.084053] LustreError: 103459:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1598.086799] LustreError: 103459:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1598.110706] Lustre: server umount MGS complete [ 1601.901534] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 19:32:48 (1760139168) [ 1608.362964] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1609.632860] Key type lgssc unregistered [ 1609.787400] LNet: 104792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.789733] LNetError: 104792:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1609.797498] LNet: Removed LNI 192.168.203.145@tcp [ 1610.162251] Key type .llcrypt unregistered [ 1610.163286] Key type ._llcrypt unregistered [ 1617.360623] Key type ._llcrypt registered [ 1617.361938] Key type .llcrypt registered [ 1617.404925] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1617.793319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1617.863462] alg: No test for adler32 (adler32-zlib) [ 1618.724953] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1618.820593] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1620.408152] Key type lgssc registered [ 1620.769670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1623.768148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1623.775675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1624.997103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1626.698109] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1630.029737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1630.184351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1631.560589] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1633.341431] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1634.493130] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1636.780580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1636.908299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1639.189708] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1641.368171] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1649.196797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1655.265202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1655.268078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1655.272442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1659.364127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1659.366557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1659.367125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1664.482748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1664.486033] Lustre: Skipped 2 previous similar messages [ 1669.088225] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1669.111970] LustreError: 108241:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1669.183342] Lustre: server umount lustre-OST0000 complete [ 1671.137152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1671.141577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1671.148510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1674.722899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1676.799782] LustreError: 108445:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1676.801572] LustreError: 108445:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1676.833732] Lustre: server umount lustre-MDT0000 complete [ 1678.307131] LustreError: 108142:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139246 with bad export cookie 5017583232269016617 [ 1678.312615] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1678.313534] LustreError: 108142:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1678.410267] LustreError: 108647:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1678.412640] LustreError: 108647:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1678.504912] Lustre: server umount lustre-MDT0001 complete [ 1682.765953] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1683.979312] Key type lgssc unregistered [ 1684.135539] LNet: 109178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.138277] LNetError: 109178:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.144678] LNet: Removed LNI 192.168.203.145@tcp [ 1684.484880] Key type .llcrypt unregistered [ 1684.486041] Key type ._llcrypt unregistered [ 1691.461438] Key type ._llcrypt registered [ 1691.463044] Key type .llcrypt registered [ 1691.498753] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1691.902503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1691.908508] alg: No test for adler32 (adler32-zlib) [ 1692.764578] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1692.838577] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1694.416197] Key type lgssc registered [ 1694.976983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1697.714791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1697.718049] LustreError: 110219:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1697.718446] Lustre: Skipped 1 previous similar message [ 1697.725156] LustreError: 110218:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1697.727576] LustreError: 110219:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 1697.732707] LustreError: 110219:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1697.738984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.740009] LustreError: 110219:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1698.929920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1702.050574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1702.152443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1703.564481] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1705.122973] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing lsmod [ 1706.050958] LustreError: 111393:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1706.053039] LustreError: 111394:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1706.058611] LustreError: 111393:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1706.060925] LustreError: 111393:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1706.064800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1706.065174] LustreError: 111393:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1706.210378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1707.489093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1707.492915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1707.498687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1711.588897] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1711.589393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1711.597238] Lustre: Skipped 1 previous similar message [ 1711.601463] Lustre: Skipped 1 previous similar message [ 1713.538887] LustreError: 111579:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1713.567940] Lustre: server umount lustre-MDT0000 complete [ 1714.778965] LustreError: 110235:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139282 with bad export cookie 774780506754843006 [ 1714.780043] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1714.782241] LustreError: 110235:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1714.785250] Lustre: 110250:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.145@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 1715.680223] LustreError: 110690:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000fd7437d2 x1845639784708992/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1715.689140] LustreError: 111782:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1715.690983] LustreError: 111782:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1715.756907] Lustre: server umount lustre-MDT0001 complete [ 1727.156707] LustreError: 111982:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1727.159772] LustreError: 111982:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1727.225523] Lustre: server umount lustre-OST0000 complete [ 1730.575063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.755737] LustreError: 112459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1730.763562] LustreError: 112459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1730.785975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1732.392361] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1735.428861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1735.556339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1736.722449] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1738.681911] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1740.358256] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1743.617376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1743.769294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1745.553962] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1747.095239] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1756.078843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1761.254038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1761.254983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1761.267567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1761.269335] Lustre: Skipped 1 previous similar message [ 1765.424155] LustreError: 114725:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1765.427836] LustreError: 114725:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1765.518274] Lustre: server umount lustre-OST0000 complete [ 1771.491803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1771.493949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1771.494848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1771.497984] LustreError: Skipped 1 previous similar message [ 1771.504707] Lustre: Skipped 2 previous similar messages [ 1773.193047] LustreError: 114927:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1773.197924] LustreError: 114927:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1773.232979] Lustre: server umount lustre-MDT0000 complete [ 1775.032222] LustreError: 114078:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139342 with bad export cookie 774780506754843685 [ 1775.037462] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1775.038915] LustreError: 114078:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1775.217961] Lustre: server umount lustre-MDT0001 complete [ 1779.815694] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1781.027525] Key type lgssc unregistered [ 1781.162383] LNet: 115660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1781.164514] LNetError: 115660:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1781.173339] LNet: Removed LNI 192.168.203.145@tcp [ 1781.489221] Key type .llcrypt unregistered [ 1781.491115] Key type ._llcrypt unregistered [ 1792.048849] Key type ._llcrypt registered [ 1792.050639] Key type .llcrypt registered [ 1792.083847] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1792.449146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1792.505646] alg: No test for adler32 (adler32-zlib) [ 1793.363570] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1793.444631] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1795.024255] Key type lgssc registered [ 1795.649594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1799.417260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1799.428846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1800.642490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.098297] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1805.530220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1805.739299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1807.595843] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1810.086545] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1811.830338] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1815.059497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1815.214515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1817.516639] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1819.553425] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1825.447292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1832.379737] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1833.329143] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1836.004142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1836.007205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1836.013247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1836.015416] Lustre: Skipped 1 previous similar message [ 1839.010719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1840.496476] LustreError: 119404:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1840.594074] Lustre: server umount lustre-OST0000 complete [ 1844.196109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1844.197439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1844.205195] Lustre: Skipped 2 previous similar messages [ 1844.208951] Lustre: Skipped 1 previous similar message [ 1848.650228] LustreError: 119609:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1848.652205] LustreError: 119609:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1848.690533] Lustre: server umount lustre-MDT0000 complete [ 1849.314206] LustreError: 117037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1849.328646] LustreError: 117037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1850.412448] LustreError: 117020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139418 with bad export cookie 6606992831460780546 [ 1850.417875] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1850.418791] LustreError: 117020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1850.500352] LustreError: 119812:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1850.504034] LustreError: 119812:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1850.601106] Lustre: server umount lustre-MDT0001 complete [ 1855.188559] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1856.750405] Key type lgssc unregistered [ 1856.919879] LNet: 120344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1856.925223] LNetError: 120344:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1856.937620] LNet: Removed LNI 192.168.203.145@tcp [ 1857.467123] Key type .llcrypt unregistered [ 1857.468345] Key type ._llcrypt unregistered [ 1865.863260] Key type ._llcrypt registered [ 1865.864152] Key type .llcrypt registered [ 1865.903737] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1866.326875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1866.334515] alg: No test for adler32 (adler32-zlib) [ 1867.237048] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1867.337962] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1868.944286] Key type lgssc registered [ 1869.602321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1873.501947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1873.515466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1874.727085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1876.511052] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1880.480107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1880.680166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1882.644916] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1885.270262] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1886.947384] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1890.213731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1890.428625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1893.135536] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1895.757681] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1899.628327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1905.126536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.128952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1905.132269] Lustre: Skipped 1 previous similar message [ 1905.133522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1905.135634] Lustre: Skipped 1 previous similar message [ 1906.548139] LustreError: 123772:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1906.645300] Lustre: server umount lustre-OST0000 complete [ 1910.242842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1910.243784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1910.249990] Lustre: Skipped 1 previous similar message [ 1910.607717] LustreError: 123974:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1910.612814] LustreError: 123974:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1910.670401] Lustre: server umount lustre-MDT0000 complete [ 1912.560171] LustreError: 123143:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139480 with bad export cookie 2676939425973693812 [ 1912.566871] LustreError: 123143:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1912.569055] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1912.656532] LustreError: 124177:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1912.659109] LustreError: 124177:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1912.785642] Lustre: server umount lustre-MDT0001 complete [ 1917.644832] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1919.291805] Key type lgssc unregistered [ 1919.471836] LNet: 124708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.477787] LNetError: 124708:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.489751] LNet: Removed LNI 192.168.203.145@tcp [ 1920.014722] Key type .llcrypt unregistered [ 1920.016870] Key type ._llcrypt unregistered [ 1930.702343] Key type ._llcrypt registered [ 1930.704868] Key type .llcrypt registered [ 1930.766945] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1931.189801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1931.266935] alg: No test for adler32 (adler32-zlib) [ 1932.142755] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1932.239403] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1933.824291] Key type lgssc registered [ 1934.440939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1937.755173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1937.767869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.992885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.915294] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1944.608801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1944.729621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1946.188532] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1948.236144] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1949.936702] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1953.157204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1953.311082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.675035] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1958.102463] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1963.563441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 1970.949258] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1972.175923] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1972.180063] Lustre: Skipped 2 previous similar messages [ 1974.197855] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1974.205284] Lustre: Skipped 2 previous similar messages [ 1976.830368] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1976.845273] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1979.557903] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.45@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1979.565229] Lustre: Skipped 2 previous similar messages [ 1985.544975] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.45@tcp 192.168.0.[2,10]@tcp [ 1987.378893] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.45/24@tcp in log lustre-MDT0000 [ 1987.384130] Lustre: Skipped 2 previous similar messages [ 1991.113037] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.45/24@tcp [ 1991.116410] Lustre: Skipped 1 previous similar message [ 1994.210651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1994.211526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.216733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1994.218155] Lustre: Skipped 1 previous similar message [ 1999.333554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1999.335803] Lustre: Skipped 1 previous similar message [ 1999.538160] LustreError: 129805:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1999.638245] Lustre: server umount lustre-OST0000 complete [ 2004.451731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.452731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2004.458306] Lustre: Skipped 2 previous similar messages [ 2004.465447] Lustre: Skipped 1 previous similar message [ 2007.691534] LustreError: 130006:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2007.693912] LustreError: 130006:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2007.723831] Lustre: server umount lustre-MDT0000 complete [ 2009.413564] LustreError: 126068:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139577 with bad export cookie 15923639251263996837 [ 2009.419624] LustreError: 126068:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2009.422664] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2009.500893] LustreError: 130208:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2009.502990] LustreError: 130208:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2009.603062] Lustre: server umount lustre-MDT0001 complete [ 2014.293071] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2015.792580] Key type lgssc unregistered [ 2015.960545] LNet: 130739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2015.962980] LNetError: 130739:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2015.976485] LNet: Removed LNI 192.168.203.145@tcp [ 2016.347977] Key type .llcrypt unregistered [ 2016.350131] Key type ._llcrypt unregistered [ 2028.333541] Key type ._llcrypt registered [ 2028.335482] Key type .llcrypt registered [ 2028.369742] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2028.666973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2028.687042] alg: No test for adler32 (adler32-zlib) [ 2029.534400] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 2029.617044] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2031.200248] Key type lgssc registered [ 2031.739502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2035.004964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2035.021949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.217130] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2036.222205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2036.225929] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.45/24@tcp [ 2037.561218] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2041.381751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2041.498868] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2041.506406] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.45/24@tcp [ 2041.524891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2043.231588] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2045.528538] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2046.828215] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2049.871298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2049.978988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2052.467620] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2054.985405] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2061.241624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2071.525823] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.527433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2071.529273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2071.532707] Lustre: Skipped 1 previous similar message [ 2075.442179] LustreError: 134580:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2075.532870] Lustre: server umount lustre-OST0000 complete [ 2077.664794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2077.666946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2077.671630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2077.674016] Lustre: Skipped 1 previous similar message [ 2081.761105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2081.761634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.281276] LustreError: 134781:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2083.284821] LustreError: 134781:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2083.320697] Lustre: server umount lustre-MDT0000 complete [ 2085.123325] LustreError: 132240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139652 with bad export cookie 13707676617241602016 [ 2085.128882] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2085.130537] LustreError: 132240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2085.218313] LustreError: 134984:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2085.220841] LustreError: 134984:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2085.302280] Lustre: server umount lustre-MDT0001 complete [ 2089.335410] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2090.649677] Key type lgssc unregistered [ 2090.786563] LNet: 135516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2090.789888] LNetError: 135516:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.799490] LNet: Removed LNI 192.168.203.145@tcp [ 2091.284696] Key type .llcrypt unregistered [ 2091.286780] Key type ._llcrypt unregistered [ 2100.463193] Key type ._llcrypt registered [ 2100.464992] Key type .llcrypt registered [ 2100.523192] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 2107.495290] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2107.825186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2107.865341] alg: No test for adler32 (adler32-zlib) [ 2108.732335] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 2108.819409] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2110.416138] Key type lgssc registered [ 2110.991576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2115.407756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2118.081698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2120.234196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2123.026610] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2126.032588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2126.059498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2126.067031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.174176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2127.186643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2127.227930] Lustre: lustre-MDT0000: new disk, initializing [ 2127.257975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2127.265267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2128.851599] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2134.315779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2134.345159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2134.399678] Lustre: 138864:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2134.447970] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2134.451302] Lustre: Skipped 1 previous similar message [ 2134.500902] Lustre: lustre-MDT0001: new disk, initializing [ 2134.555616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2134.577168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2134.583554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2136.262411] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2139.473884] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2141.125585] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2143.709965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2143.753855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2143.902963] Lustre: lustre-OST0000: new disk, initializing [ 2143.906089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2143.951423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2146.185121] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2148.857714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2148.860564] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2148.892727] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2149.139847] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2150.441903] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2150.529706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2152.224371] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2152.333107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2153.954624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2153.956542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2153.958089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2153.958261] LustreError: Skipped 1 previous similar message [ 2153.964214] Lustre: Skipped 1 previous similar message [ 2155.311195] LustreError: 141801:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2155.409128] Lustre: server umount lustre-OST0000 complete [ 2159.076772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.078194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.084036] Lustre: Skipped 1 previous similar message [ 2159.092917] Lustre: Skipped 2 previous similar messages [ 2164.194180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2164.198541] Lustre: Skipped 1 previous similar message [ 2169.315223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2169.321617] Lustre: Skipped 1 previous similar message [ 2171.360222] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2171.410147] LustreError: 142004:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2171.414133] LustreError: 142004:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2171.459349] Lustre: server umount lustre-MDT0000 complete [ 2172.989477] LustreError: 140609:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139740 with bad export cookie 7643634414816489640 [ 2172.993772] LustreError: 140609:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2172.997985] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2173.072152] LustreError: 142207:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2173.076238] LustreError: 142207:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2173.167617] Lustre: server umount lustre-MDT0001 complete [ 2177.121086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.371214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2179.147813] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2182.626071] LustreError: 142683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.635157] LustreError: 142683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2183.205323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2183.378714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2185.255311] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2187.830941] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2189.192298] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2192.266508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2192.401265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.892430] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2197.265762] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2198.983093] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2203.242779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2204.620474] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2204.744392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2209.068979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2209.110093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2209.192756] Lustre: lustre-OST0001: new disk, initializing [ 2209.195358] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2209.240345] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2211.676547] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2215.494896] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2218.533384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2218.539718] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2218.590112] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2218.702665] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2220.478613] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2220.566440] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2221.873989] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff990285c7b000.ost_server_uuid 50 [ 2222.600573] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff990285c7b000.ost_server_uuid in FULL state after 0 sec [ 2228.716236] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2228.717552] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.720566] LustreError: Skipped 1 previous similar message [ 2228.722057] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2228.727341] Lustre: Skipped 1 previous similar message [ 2231.283309] LustreError: 146977:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2231.287122] LustreError: 146977:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2231.317587] Lustre: server umount lustre-OST0001 complete [ 2233.312997] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2233.315250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.826749] LustreError: 144280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.834687] LustreError: 144280:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2238.948152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2238.949170] LustreError: 144281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.949952] Lustre: Skipped 4 previous similar messages [ 2238.954439] LustreError: 144281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2239.089132] LustreError: 147179:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2239.093058] LustreError: 147179:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2239.181768] Lustre: server umount lustre-OST0000 complete [ 2244.068873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2244.074517] Lustre: Skipped 2 previous similar messages [ 2255.328195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2255.387038] LustreError: 147383:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2255.393286] LustreError: 147383:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2255.429454] Lustre: server umount lustre-MDT0000 complete [ 2256.924176] LustreError: 145715:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139824 with bad export cookie 7643634414816490522 [ 2256.927424] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2256.931594] LustreError: 145715:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2257.079716] Lustre: server umount lustre-MDT0001 complete [ 2265.114209] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2266.367194] Key type lgssc unregistered [ 2266.505698] LNet: 148574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2266.510508] LNetError: 148574:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2266.521581] LNet: Removed LNI 192.168.203.145@tcp [ 2266.953833] Key type .llcrypt unregistered [ 2266.955833] Key type ._llcrypt unregistered [ 2281.564801] Key type ._llcrypt registered [ 2281.565978] Key type .llcrypt registered [ 2281.601586] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 2288.678772] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2289.020417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2289.041336] alg: No test for adler32 (adler32-zlib) [ 2289.895634] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 2289.991734] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2291.576173] Key type lgssc registered [ 2292.134641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2296.466563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2299.726589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2302.582383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2305.632528] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2308.860621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2308.891857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2308.902208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2309.994734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2310.005835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2310.048363] Lustre: lustre-MDT0000: new disk, initializing [ 2310.079688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2310.089444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2311.813236] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2317.315623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2317.351486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2317.390742] Lustre: 152499:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2317.425541] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2317.429041] Lustre: Skipped 1 previous similar message [ 2317.476741] Lustre: lustre-MDT0001: new disk, initializing [ 2317.526174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2317.549901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2317.556587] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2319.343119] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2323.476584] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2325.249594] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2328.561088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2328.606106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2328.757632] Lustre: lustre-OST0000: new disk, initializing [ 2328.761357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2328.809287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2330.688209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2330.692782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2330.739432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2331.247795] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2335.138185] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2336.726109] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2336.795783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2338.511899] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2338.632941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2340.833542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2340.838097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.846247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2341.348070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2341.349243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2341.355270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2345.652743] LustreError: 155434:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2345.745341] Lustre: server umount lustre-OST0000 complete [ 2348.513761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2348.518409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2348.526744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2348.532185] Lustre: Skipped 1 previous similar message [ 2351.586766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2351.587426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2353.675917] LustreError: 155636:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2353.679660] LustreError: 155636:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2353.719983] Lustre: server umount lustre-MDT0000 complete [ 2355.546081] LustreError: 153278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139923 with bad export cookie 9056944793515325855 [ 2355.552334] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2355.552967] LustreError: 153278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2355.646422] LustreError: 155838:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2355.649705] LustreError: 155838:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2355.744350] Lustre: server umount lustre-MDT0001 complete [ 2359.908737] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 19:45:26 (1760139926) [ 2367.335315] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 2371.614986] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2376.086589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2379.353334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2382.093862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2385.140781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2388.429931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2388.472704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2388.602560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2388.621518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2388.671244] Lustre: lustre-MDT0000: new disk, initializing [ 2388.709983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2388.721408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2390.432427] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2395.825070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2395.859883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2395.915345] Lustre: 158824:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2395.947745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2395.950011] Lustre: Skipped 1 previous similar message [ 2395.990892] Lustre: lustre-MDT0001: new disk, initializing [ 2396.028292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2396.042751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2396.046589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2397.684589] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2401.532180] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2403.203739] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2406.240952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2406.270958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2406.385711] Lustre: lustre-OST0000: new disk, initializing [ 2406.388178] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2406.422832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2408.389292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2408.393094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2408.431902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2408.750425] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2412.824324] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2415.291043] Lustre: Failing over lustre-OST0000 [ 2415.339214] LustreError: 161266:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2415.342920] LustreError: 161266:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2415.431763] Lustre: server umount lustre-OST0000 complete [ 2429.509382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2429.612086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.615459] Lustre: Skipped 1 previous similar message [ 2429.619619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2429.626230] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2432.089800] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2433.935545] Lustre: Failing over lustre-MDT0000 [ 2434.015952] LustreError: 162238:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2434.018177] LustreError: 162238:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2434.045852] Lustre: server umount lustre-MDT0000 complete [ 2434.488725] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2434.721151] Lustre: 150715:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760139986/real 1760139986] req@000000006ab42068 x1845640410754048/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1760140002 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2439.595876] LustreError: 161179:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2439.605821] LustreError: 161179:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2440.160367] Lustre: 150715:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760139991/real 1760139991] req@000000008d5ad9ec x1845640410755072/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1760140007 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2440.161811] LustreError: 158834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.174151] Lustre: 150715:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2440.182397] LustreError: 158834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2444.320170] Lustre: 150715:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760139996/real 1760139996] req@0000000036fe8c35 x1845640410756096/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1760140012 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2444.715438] LustreError: 158829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.036167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2448.107614] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.115372] Lustre: 158827:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.145@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2448.225823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2448.298521] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2449.835526] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2449.933048] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2453.481617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2453.493495] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2453.498813] Lustre: Skipped 1 previous similar message [ 2453.526052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2458.595863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2458.597767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2458.600034] LustreError: Skipped 1 previous similar message [ 2458.601343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2458.608068] Lustre: Skipped 4 previous similar messages [ 2458.616232] Lustre: Skipped 2 previous similar messages [ 2460.978165] LustreError: 163121:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2460.982890] LustreError: 163121:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2461.067984] Lustre: server umount lustre-OST0000 complete [ 2468.833539] LustreError: 160166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.840962] LustreError: 160166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2468.847347] Lustre: server umount lustre-MDT0000 complete [ 2470.522397] LustreError: 158815:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140038 with bad export cookie 9056944793515327934 [ 2470.527904] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2470.529181] LustreError: 158815:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2470.724190] Lustre: server umount lustre-MDT0001 complete [ 2475.611266] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2477.005542] Key type lgssc unregistered [ 2477.141678] LNet: 164055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2477.146410] LNetError: 164055:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2477.158531] LNet: Removed LNI 192.168.203.145@tcp [ 2477.611219] Key type .llcrypt unregistered [ 2477.613456] Key type ._llcrypt unregistered [ 2500.268254] Key type ._llcrypt registered [ 2500.269696] Key type .llcrypt registered [ 2500.312707] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2500.686369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2500.717705] alg: No test for adler32 (adler32-zlib) [ 2501.581821] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 2501.671748] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2503.248167] Key type lgssc registered [ 2503.772457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2507.262456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2507.272069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2508.455447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2510.034184] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2514.028873] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2516.415407] LustreError: 165584:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2516.421971] LustreError: 165584:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2516.427590] LustreError: 165584:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 2516.524296] LustreError: 166106:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2516.634634] Lustre: server umount lustre-MDT0000 complete [ 2522.799581] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 2526.259077] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2529.861144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2531.987212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2533.828633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2535.853207] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2538.021317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2538.050707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2538.143673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2538.154415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2538.192868] Lustre: lustre-MDT0000: new disk, initializing [ 2538.217943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2538.223624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2539.443238] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2543.736129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2543.763913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2543.801040] Lustre: 168746:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2543.843247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2543.845393] Lustre: Skipped 1 previous similar message [ 2543.885987] Lustre: lustre-MDT0001: new disk, initializing [ 2543.934394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2543.951793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2543.959838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2545.478294] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2549.177844] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2550.714298] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2553.045973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2553.069410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2553.152870] Lustre: lustre-OST0000: new disk, initializing [ 2553.154487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2553.180591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2554.976311] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2556.479959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2556.484392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2556.519179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2557.982049] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2559.720094] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2560.216423] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 19:48:47 (1760140127) [ 2615.776575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2615.778608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.781649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2617.824383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2617.824639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2617.825974] Lustre: Skipped 1 previous similar message [ 2617.828906] Lustre: Skipped 1 previous similar message [ 2620.536454] LustreError: 172244:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2620.538270] LustreError: 172244:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2620.575212] Lustre: server umount lustre-MDT0000 complete [ 2621.648394] LustreError: 168736:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140189 with bad export cookie 8163366119200648619 [ 2621.649334] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2621.651455] LustreError: 168736:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2622.944386] LustreError: 170502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2622.944483] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2622.948369] LustreError: 170502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2622.957106] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2626.976404] LustreError: 170502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.976445] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2627.770201] LustreError: 172449:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2627.771796] LustreError: 172449:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2627.830608] Lustre: server umount lustre-MDT0001 complete [ 2635.050964] LustreError: 172650:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2635.052602] LustreError: 172650:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2635.104323] Lustre: server umount lustre-OST0000 complete [ 2637.744071] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 2640.683890] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2645.118836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2648.328428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2651.297412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2653.876181] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2656.825750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2656.865433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2656.999023] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2657.018929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2657.073983] Lustre: lustre-MDT0000: new disk, initializing [ 2657.103088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.110689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2658.455336] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2663.214413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2663.253778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2663.303177] Lustre: 174985:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2663.335993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2663.339799] Lustre: Skipped 1 previous similar message [ 2663.388076] Lustre: lustre-MDT0001: new disk, initializing [ 2663.448273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2663.454304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2665.268307] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2669.187174] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2670.948383] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2674.079819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2674.112373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2674.238936] Lustre: lustre-OST0000: new disk, initializing [ 2674.241505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2674.273194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2674.275156] Lustre: Skipped 1 previous similar message [ 2676.654446] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2679.809686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2679.813882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2679.863408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2680.463630] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2682.206285] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2682.343162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2684.169319] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2684.291726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2690.021357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2690.021927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.025246] LustreError: Skipped 1 previous similar message [ 2690.032301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2690.036758] Lustre: Skipped 1 previous similar message [ 2691.182095] LustreError: 177925:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2691.184931] LustreError: 177925:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2691.250960] Lustre: server umount lustre-OST0000 complete [ 2694.113219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2694.115931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.120266] Lustre: Skipped 1 previous similar message [ 2698.611618] Lustre: server umount lustre-MDT0000 complete [ 2700.220203] LustreError: 174976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140268 with bad export cookie 8163366119200807435 [ 2700.226413] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2700.227156] LustreError: 174976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2700.336189] LustreError: 178328:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2700.340083] LustreError: 178328:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2700.443299] Lustre: server umount lustre-MDT0001 complete [ 2704.195540] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:51:10 (1760140270) [ 2710.725225] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 2714.388528] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2717.810229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2720.590601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2722.626296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2725.165818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2728.275292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2728.305793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.399194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2728.409843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2728.449272] Lustre: lustre-MDT0000: new disk, initializing [ 2728.473552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2728.479858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2729.929201] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2734.761834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2734.789754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2734.824295] Lustre: 181311:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2734.942682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2736.468579] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2740.524907] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2741.992791] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2744.865871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2744.904128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2747.196263] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2750.976818] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2752.507141] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2752.538373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2762.723455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2762.724287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.724892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2762.724900] Lustre: Skipped 3 previous similar messages [ 2762.725706] LustreError: Skipped 1 previous similar message [ 2762.729481] Lustre: Skipped 2 previous similar messages [ 2764.205068] LustreError: 183859:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2764.207243] LustreError: 183859:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2764.265342] Lustre: server umount lustre-OST0000 complete [ 2771.547185] Lustre: server umount lustre-MDT0000 complete [ 2772.769369] LustreError: 181303:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140340 with bad export cookie 8163366119200808289 [ 2772.773622] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2772.916348] Lustre: server umount lustre-MDT0001 complete [ 2775.288979] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:52:22 (1760140342) [ 2781.282043] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 2784.663237] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2787.631229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2789.718761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2791.734789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2793.705110] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2795.785776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2795.809657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.895937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2795.906813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2795.908900] Lustre: Skipped 3 previous similar messages [ 2795.946417] Lustre: lustre-MDT0000: new disk, initializing [ 2795.947772] Lustre: Skipped 2 previous similar messages [ 2795.972584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2795.974387] Lustre: Skipped 2 previous similar messages [ 2795.980562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2795.982572] Lustre: Skipped 2 previous similar messages [ 2797.168289] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2801.093883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2801.113607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2801.145439] Lustre: 187248:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2801.239950] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2802.385693] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2805.340700] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2806.504705] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2808.626385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2808.646270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2810.352418] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2812.244777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2813.156809] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2817.506527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2817.507507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2817.508583] LustreError: Skipped 2 previous similar messages [ 2817.509050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2817.512573] Lustre: Skipped 3 previous similar messages [ 2817.513838] Lustre: Skipped 4 previous similar messages [ 2820.908120] LustreError: 189794:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2820.909739] LustreError: 189794:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2820.963846] Lustre: server umount lustre-OST0000 complete [ 2828.181778] Lustre: server umount lustre-MDT0000 complete [ 2829.281168] LustreError: 187239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140397 with bad export cookie 8163366119200809360 [ 2829.283729] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2829.284207] LustreError: 187239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2829.414974] Lustre: server umount lustre-MDT0001 complete [ 2832.692761] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2833.640615] Key type lgssc unregistered [ 2833.760414] LNet: 190727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2833.764113] LNetError: 190727:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2833.773299] LNet: Removed LNI 192.168.203.145@tcp [ 2834.065714] Key type .llcrypt unregistered [ 2834.066741] Key type ._llcrypt unregistered [ 2842.620733] Key type ._llcrypt registered [ 2842.621705] Key type .llcrypt registered [ 2842.656212] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2843.101269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2843.124700] alg: No test for adler32 (adler32-zlib) [ 2843.975691] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 2844.049956] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2845.624124] Key type lgssc registered [ 2845.923277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2848.200025] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2848.206516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2849.346652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2850.513562] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2853.086166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2853.198164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2854.367800] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2855.991059] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2857.062391] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2859.037270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2859.138514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2860.202695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2860.795616] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2862.398733] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2875.872772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2875.874736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2875.877942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2878.947783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2878.948656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2878.952878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2881.070069] LustreError: 194369:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2881.122188] Lustre: server umount lustre-OST0000 complete [ 2884.066656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2884.067199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2884.068163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2884.068168] Lustre: Skipped 1 previous similar message [ 2884.069811] Lustre: Skipped 1 previous similar message [ 2888.380195] LustreError: 194572:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2888.382268] LustreError: 194572:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2888.427231] Lustre: server umount lustre-MDT0000 complete [ 2889.185596] LustreError: 192102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2889.190592] LustreError: 192102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2889.811825] LustreError: 192081:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140457 with bad export cookie 4856965988886433316 [ 2889.817217] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2889.907525] LustreError: 194774:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2889.909297] LustreError: 194774:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2889.992013] Lustre: server umount lustre-MDT0001 complete [ 2893.561527] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2894.471415] Key type lgssc unregistered [ 2894.599323] LNet: 195305:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2894.601354] LNetError: 195305:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2894.613265] LNet: Removed LNI 192.168.203.145@tcp [ 2894.892043] Key type .llcrypt unregistered [ 2894.892979] Key type ._llcrypt unregistered [ 2903.103587] Key type ._llcrypt registered [ 2903.104496] Key type .llcrypt registered [ 2903.134659] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2903.421390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2903.474320] alg: No test for adler32 (adler32-zlib) [ 2904.319694] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 2904.396870] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2905.968160] Key type lgssc registered [ 2906.384397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2908.720420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2908.728084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2909.868865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2911.026434] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2913.611923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2913.723870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2914.939592] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2916.514066] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2917.616132] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2919.744769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2919.867830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2920.874531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2921.617298] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2923.278250] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2929.124532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2929.125296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2929.126300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2929.131398] Lustre: Skipped 1 previous similar message [ 2932.403175] LustreError: 198922:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2932.487221] Lustre: server umount lustre-OST0000 complete [ 2935.131777] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff990284843800.ost_server_uuid 50 [ 2936.038544] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2949.219737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.221103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2949.227058] Lustre: Skipped 1 previous similar message [ 2949.230703] Lustre: Skipped 2 previous similar messages [ 2952.252489] LustreError: 199274:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2952.255464] LustreError: 199274:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2952.282279] Lustre: server umount lustre-MDT0000 complete [ 2953.356096] LustreError: 196658:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140521 with bad export cookie 6198450142186065567 [ 2953.358840] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2953.359214] LustreError: 196658:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2953.416251] LustreError: 199476:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2953.417961] LustreError: 199476:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2953.487756] Lustre: server umount lustre-MDT0001 complete [ 2956.083497] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 19:55:23 (1760140523) [ 2958.085656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2958.218981] LustreError: 200146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.223664] LustreError: 200146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2958.238369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2959.306113] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2961.877259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2963.163973] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2964.824153] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2965.849350] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2967.759683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2967.856689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2967.858490] Lustre: Skipped 1 previous similar message [ 2968.870063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2969.412952] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2970.993765] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2972.987114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2973.009583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2973.059675] Lustre: lustre-OST0001: new disk, initializing [ 2973.061517] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2975.541328] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2977.275475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2977.278334] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2977.309059] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2978.558092] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2982.374435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2982.375289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2982.378282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2982.381009] Lustre: Skipped 2 previous similar messages [ 2985.773090] LustreError: 203348:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2985.774808] LustreError: 203348:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2985.787614] Lustre: server umount lustre-OST0000 complete [ 2987.473932] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2987.494555] LustreError: 202661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2987.498624] LustreError: 202661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2987.541287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2988.646318] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2988.710280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2988.970991] LustreError: 201742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2992.611247] LustreError: 202661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2992.615085] LustreError: 202661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2997.732951] LustreError: 201741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.739913] LustreError: 201741:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3002.855276] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3002.856499] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3002.859786] LustreError: Skipped 1 previous similar message [ 3002.860318] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3002.860334] Lustre: Skipped 1 previous similar message [ 3002.867001] Lustre: Skipped 2 previous similar messages [ 3004.849316] LustreError: 204049:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3004.853244] LustreError: 204049:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3004.939460] Lustre: server umount lustre-OST0001 complete [ 3007.972317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3007.972506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.985328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3007.990471] Lustre: Skipped 2 previous similar messages [ 3018.209079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3018.210671] Lustre: Skipped 3 previous similar messages [ 3021.280141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3021.309500] LustreError: 204250:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3021.311197] LustreError: 204250:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3021.336043] Lustre: server umount lustre-MDT0000 complete [ 3022.305318] LustreError: 200146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3022.309212] LustreError: 200146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3022.415977] LustreError: 202652:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140590 with bad export cookie 6198450142186066624 [ 3022.419386] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3022.420027] LustreError: 202652:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3022.549741] Lustre: server umount lustre-MDT0001 complete [ 3024.206849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3025.348844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3026.465559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3027.605940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3029.582328] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 19:56:36 (1760140596) [ 3031.526028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.613156] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3031.623649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3031.686917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3031.688661] Lustre: Skipped 1 previous similar message [ 3032.815115] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3035.276271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3035.305176] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3035.315915] Lustre: 205945:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3036.684303] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3039.306970] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3040.945631] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3044.066157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3044.175729] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3046.314558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 3046.780473] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3049.288934] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3052.235262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3052.277446] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3052.415653] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3052.419662] Lustre: Skipped 2 previous similar messages [ 3052.553718] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3054.701827] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3056.969758] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3061.729113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3061.729355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3061.731260] LustreError: Skipped 1 previous similar message [ 3061.731729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3061.736809] Lustre: Skipped 2 previous similar messages [ 3063.854088] LustreError: 209141:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3063.857206] LustreError: 209141:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3063.876821] Lustre: server umount lustre-OST0000 complete [ 3067.308230] LustreError: 207576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3078.116234] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3078.118334] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3078.121294] Lustre: Skipped 1 previous similar message [ 3080.419335] Lustre: server umount lustre-OST0001 complete [ 3088.119351] Lustre: server umount lustre-MDT0000 complete [ 3089.929594] LustreError: 205937:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140657 with bad export cookie 6198450142186067982 [ 3089.936519] LustreError: 205937:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3089.938759] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3090.132369] Lustre: server umount lustre-MDT0001 complete [ 3092.573064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3093.938928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3095.405306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3097.197871] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3099.216836] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 19:57:46 (1760140666) [ 3104.013424] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 3106.916786] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3110.223939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3112.222857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3114.012942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3115.827058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3117.817520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3117.836785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.908154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3117.917494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3117.951254] Lustre: lustre-MDT0000: new disk, initializing [ 3117.973454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3117.978935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3119.143025] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3122.910695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3122.929261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3122.956481] Lustre: 213564:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3122.975911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3122.977523] Lustre: Skipped 1 previous similar message [ 3123.011329] Lustre: lustre-MDT0001: new disk, initializing [ 3123.049816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3123.052638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3124.173158] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3127.024991] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3128.657177] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3131.988690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3132.030115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3132.176968] Lustre: lustre-OST0000: new disk, initializing [ 3132.181364] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3134.588210] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3138.518887] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3139.943082] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3141.632207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3141.636741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3141.671506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3142.078571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3143.598340] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3143.663808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3146.721704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3146.722862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3146.725098] LustreError: Skipped 2 previous similar messages [ 3146.727897] Lustre: Skipped 2 previous similar messages [ 3146.728497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3146.733112] Lustre: Skipped 9 previous similar messages [ 3150.318067] LustreError: 216518:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3150.319685] LustreError: 216518:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3150.380523] Lustre: server umount lustre-OST0000 complete [ 3157.650815] Lustre: server umount lustre-MDT0000 complete [ 3158.743381] LustreError: 213556:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140726 with bad export cookie 6198450142186069753 [ 3158.746340] LustreError: 213556:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3158.747026] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3158.881466] Lustre: server umount lustre-MDT0001 complete [ 3161.366345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3161.517246] LustreError: 217396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3161.525155] LustreError: 217396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 3162.698325] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3165.499719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3166.878441] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3168.537132] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3169.613720] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3171.684371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3174.037854] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3176.026394] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3177.543196] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3177.659698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3179.339466] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3179.420058] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3181.030400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3182.415535] Lustre: server umount lustre-OST0000 complete [ 3185.051527] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3185.165001] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3186.979952] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3210.425161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3210.531403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3210.533944] Lustre: Skipped 5 previous similar messages [ 3212.213096] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3213.911427] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3214.444503] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3217.386175] LustreError: 221784:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3217.387957] LustreError: 221784:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3217.436797] Lustre: server umount lustre-OST0000 complete [ 3220.643351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3220.644199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3220.646555] Lustre: Skipped 6 previous similar messages [ 3220.649238] Lustre: Skipped 6 previous similar messages [ 3224.727407] Lustre: server umount lustre-MDT0000 complete [ 3226.081475] LustreError: 217395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3226.085722] LustreError: 217395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3226.099095] LustreError: 217377:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140793 with bad export cookie 6198450142186070621 [ 3226.103095] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3226.261686] Lustre: server umount lustre-MDT0001 complete [ 3229.292506] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 19:59:56 (1760140796) [ 3231.559817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3233.215852] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3236.241595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3237.772458] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3239.770522] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3240.911117] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3243.042947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3245.076916] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3246.802400] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3248.049206] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3258.321234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3259.530960] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3259.591682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3261.622451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3261.646608] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3261.695638] Lustre: lustre-OST0001: new disk, initializing [ 3261.698379] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3263.385092] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3266.307015] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3267.542020] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3267.579511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3267.582405] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3267.606084] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3268.628429] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3269.829855] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3269.897153] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3272.675476] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3272.679041] LustreError: Skipped 2 previous similar messages [ 3276.607944] Lustre: server umount lustre-OST0001 complete [ 3278.328562] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3278.396528] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3279.579274] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3279.642612] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3303.160965] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3305.073086] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3306.855915] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3307.407581] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3314.106195] Lustre: server umount lustre-OST0001 complete [ 3332.210051] Lustre: server umount lustre-OST0000 complete [ 3339.607929] Lustre: server umount lustre-MDT0000 complete [ 3340.805344] LustreError: 222885:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140908 with bad export cookie 6198450142186071552 [ 3340.810729] LustreError: 222885:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3340.811215] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3355.104090] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3355.140833] LustreError: 229434:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3355.142675] LustreError: 229434:0:(obd_class.h:479:obd_check_dev()) Skipped 37 previous similar messages [ 3355.228885] Lustre: server umount lustre-MDT0001 complete [ 3357.063986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3358.328323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3359.553963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3360.783392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3362.905120] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:02:09 (1760140929) [ 3364.972142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3365.056478] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3365.065029] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3365.129230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3365.131749] Lustre: Skipped 5 previous similar messages [ 3366.276180] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3369.026606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3369.055062] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3369.067947] Lustre: 230923:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3370.399529] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3371.968634] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3373.102140] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3375.210270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3375.276180] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3377.128236] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3378.808556] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3387.568891] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3387.592029] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3389.371978] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3390.945724] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3394.532484] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 3395.724208] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3395.788740] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3396.953712] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3397.027879] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3398.138843] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99028665f000.ost_server_uuid 50 [ 3398.635545] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99028665f000.ost_server_uuid in FULL state after 0 sec [ 3398.951994] Lustre: Permanently deactivating lustre-OST0001 [ 3398.956899] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3399.744828] Lustre: Permanently reactivating lustre-OST0001 [ 3403.238472] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3403.238611] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3403.243340] Lustre: Skipped 6 previous similar messages [ 3403.245323] LustreError: Skipped 2 previous similar messages [ 3403.245862] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3403.249037] Lustre: Skipped 17 previous similar messages [ 3406.708532] Lustre: server umount lustre-OST0001 complete [ 3408.355943] LustreError: 233116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3408.364197] LustreError: 233116:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 3422.176076] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3422.239166] Lustre: server umount lustre-OST0000 complete [ 3429.529527] Lustre: server umount lustre-MDT0000 complete [ 3430.719472] LustreError: 230961:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140998 with bad export cookie 6198450142186072812 [ 3430.723530] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3430.884110] Lustre: server umount lustre-MDT0001 complete [ 3432.677745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3433.972889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3435.175070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3436.337719] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3438.295942] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:03:25 (1760141005) [ 3438.605586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3440.567899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3440.651933] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3440.660732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3440.663492] Lustre: Skipped 5 previous similar messages [ 3441.816965] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3444.304235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3444.337094] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3444.345450] Lustre: 237065:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3445.543225] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3447.073838] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3448.069035] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3450.005628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3450.025107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3450.094483] Lustre: Permanently deactivating lustre-OST0000 [ 3450.098509] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3451.762300] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3453.274483] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3454.960598] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3455.419038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3457.219252] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3458.842486] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3461.152852] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3464.494251] Lustre: Permanently reactivating lustre-OST0000 [ 3467.700093] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3467.702347] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3467.705516] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3467.720270] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3467.726625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3476.153842] Lustre: server umount lustre-OST0001 complete [ 3479.395495] Lustre: server umount lustre-OST0000 complete [ 3486.669932] Lustre: server umount lustre-MDT0000 complete [ 3487.663912] LustreError: 239607:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760141055 with bad export cookie 6198450142186074450 [ 3487.666430] LustreError: 239607:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3487.666447] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3487.783469] Lustre: server umount lustre-MDT0001 complete [ 3490.910868] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3491.788125] Key type lgssc unregistered [ 3491.913339] LNet: 241456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3491.915339] LNetError: 241456:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3491.927283] LNet: Removed LNI 192.168.203.145@tcp [ 3492.201938] Key type .llcrypt unregistered [ 3492.202953] Key type ._llcrypt unregistered [ 3499.880358] Key type ._llcrypt registered [ 3499.881317] Key type .llcrypt registered [ 3499.918186] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3500.274559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3500.330036] alg: No test for adler32 (adler32-zlib) [ 3501.179303] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 3501.255926] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3502.824143] Key type lgssc registered [ 3503.142970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3504.136873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3506.162812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3506.169341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3507.350149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3508.869518] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3511.835118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3511.879964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3511.931697] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3511.946442] Lustre: 242868:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3511.951519] Lustre: Permanently deactivating lustre-MDT0001 [ 3511.959502] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3512.131964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3513.855265] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3516.478393] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3517.433447] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3518.367747] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3521.413434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3521.577983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3524.182105] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3526.141155] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3528.756385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3528.824307] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3530.755385] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3532.917813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3532.918415] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3533.057177] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3534.707112] Lustre: Permanently reactivating lustre-MDT0001 [ 3534.715942] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3534.717527] Lustre: Skipped 2 previous similar messages [ 3537.016141] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3537.016241] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3540.503464] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3540.511340] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3540.513862] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3540.521402] LustreError: 246005:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3540.522226] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3540.526799] LustreError: 246005:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3540.535464] LustreError: 246005:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.145@tcp: cfg command failed: rc = -108 [ 3540.540397] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3540.540397] [ 3540.544867] LustreError: 242872:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3542.264281] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3542.746368] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3544.319314] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3544.397408] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3545.628803] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3545.701780] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3546.437924] Lustre: Permanently deactivating lustre-MDT0001 [ 3546.443502] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3546.445671] Lustre: Skipped 2 previous similar messages [ 3552.250669] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3557.858774] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3557.860675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3557.863798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3560.929432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3560.930364] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3560.934908] Lustre: Skipped 2 previous similar messages [ 3562.303287] LustreError: 246993:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3562.332050] Lustre: server umount lustre-MDT0000 complete [ 3563.528087] LustreError: 245984:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760141131 with bad export cookie 3555100458916604985 [ 3563.531587] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3563.532136] LustreError: 245984:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3563.599187] LustreError: 247194:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3563.601908] LustreError: 247194:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3563.674975] Lustre: server umount lustre-MDT0001 complete [ 3570.930197] LustreError: 247394:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3570.931973] LustreError: 247394:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3570.943707] Lustre: server umount lustre-OST0000 complete [ 3574.321980] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:05:41 (1760141141) [ 3577.564215] LustreError: 248401:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3577.565984] LustreError: 248401:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3577.628813] Lustre: server umount lustre-OST0001 complete [ 3579.469540] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 3582.043194] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3585.069745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3587.084595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3588.943363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3590.996292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3593.219444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3593.243541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.326523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3593.329167] Lustre: Skipped 2 previous similar messages [ 3593.337915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3593.376154] Lustre: lustre-MDT0000: new disk, initializing [ 3593.401817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3593.407918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3594.675309] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3598.672433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3598.697166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3598.730687] Lustre: 250581:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3598.754540] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3598.756820] Lustre: Skipped 1 previous similar message [ 3598.791863] Lustre: lustre-MDT0001: new disk, initializing [ 3598.836932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3598.840555] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3600.034814] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3602.991490] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3604.073639] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3606.172234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3606.198647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3606.290725] Lustre: lustre-OST0000: new disk, initializing [ 3606.292471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3606.323259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3606.324954] Lustre: Skipped 1 previous similar message [ 3608.069158] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3608.569872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3608.572878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3608.611245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3611.009128] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3611.617457] LustreError: 251826:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3615.656872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3615.685427] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3615.742953] Lustre: lustre-OST0001: new disk, initializing [ 3615.744538] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3617.554050] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3620.519572] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3621.079583] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3621.704140] LustreError: 251826:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3621.711594] LustreError: 251826:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3624.958474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3624.961529] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3624.988958] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3631.800063] LustreError: 251826:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3633.073074] Lustre: lustre-OST0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 3633.074831] Lustre: Skipped 3 previous similar messages [ 3635.172987] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3635.173858] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3635.175202] LustreError: Skipped 1 previous similar message [ 3635.178791] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3635.181273] Lustre: Skipped 1 previous similar message [ 3638.573153] LustreError: 254096:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3638.575233] LustreError: 254096:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3638.590762] Lustre: server umount lustre-OST0001 complete [ 3640.292046] LustreError: 252753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3640.295907] LustreError: 252753:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3645.412418] LustreError: 252314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3645.416759] LustreError: 252314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3650.530625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.530989] LustreError: 254212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3650.531674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3650.531681] Lustre: Skipped 1 previous similar message [ 3650.533753] Lustre: Skipped 2 previous similar messages [ 3650.541342] LustreError: 254212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3655.651881] LustreError: 252312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3655.656010] LustreError: 252312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3656.427360] LustreError: 254319:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3656.430458] LustreError: 254319:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3656.495553] Lustre: server umount lustre-OST0000 complete [ 3660.256913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3660.258941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3660.262355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3660.263858] Lustre: Skipped 3 previous similar messages [ 3663.952081] Lustre: server umount lustre-MDT0000 complete [ 3665.075079] LustreError: 252328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760141232 with bad export cookie 3555100458916607792 [ 3665.078155] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3665.080634] LustreError: 252328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3671.239977] Lustre: server umount lustre-MDT0001 complete [ 3674.464829] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3675.487645] Key type lgssc unregistered [ 3675.626544] LNet: 255253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3675.629857] LNetError: 255253:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3675.639464] LNet: Removed LNI 192.168.203.145@tcp [ 3675.937815] Key type .llcrypt unregistered [ 3675.938848] Key type ._llcrypt unregistered [ 3685.976865] Key type ._llcrypt registered [ 3685.977806] Key type .llcrypt registered [ 3686.011388] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 3691.696963] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3692.026146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3692.071148] alg: No test for adler32 (adler32-zlib) [ 3692.909642] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 3692.985530] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3694.560115] Key type lgssc registered [ 3694.927304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3697.728626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3699.686235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3701.474962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3703.422769] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3705.517802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3705.539318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3705.545961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3706.627712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3706.637464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3706.673608] Lustre: lustre-MDT0000: new disk, initializing [ 3706.695762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3706.701204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3707.832594] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3711.592153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3711.614921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3711.643320] Lustre: 259174:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3711.663157] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3711.664720] Lustre: Skipped 1 previous similar message [ 3711.698450] Lustre: lustre-MDT0001: new disk, initializing [ 3711.728788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3711.739452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3711.742529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3712.915948] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3715.792066] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3716.831287] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3718.930365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3718.947731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3719.035221] Lustre: lustre-OST0000: new disk, initializing [ 3719.036925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3719.062123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3720.710617] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3721.526166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3721.528844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3721.559043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3723.649909] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3724.789706] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3724.851516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3725.955717] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3726.017671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3726.818939] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3726.819099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3726.821475] LustreError: Skipped 1 previous similar message [ 3726.824969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3726.827297] Lustre: Skipped 1 previous similar message [ 3731.939400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3731.941010] Lustre: Skipped 1 previous similar message [ 3732.654101] LustreError: 262112:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3732.723072] Lustre: server umount lustre-OST0000 complete [ 3737.057627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3737.058281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3737.060730] Lustre: Skipped 2 previous similar messages [ 3737.063736] Lustre: Skipped 1 previous similar message [ 3742.177051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3742.178732] Lustre: Skipped 1 previous similar message [ 3747.297050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3747.298879] Lustre: Skipped 1 previous similar message [ 3748.320105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3748.351579] LustreError: 262315:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3748.353318] LustreError: 262315:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3748.376639] Lustre: server umount lustre-MDT0000 complete [ 3749.487074] LustreError: 259165:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760141317 with bad export cookie 13855712575806476054 [ 3749.489313] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3749.489977] LustreError: 259165:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3749.549548] LustreError: 262518:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3749.552118] LustreError: 262518:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3749.625924] Lustre: server umount lustre-MDT0001 complete [ 3752.249955] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 20:08:39 (1760141319) [ 3754.220505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3754.387048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3755.496844] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3757.996688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3759.298136] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3760.861592] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3761.890266] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3763.920308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3764.020831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3764.022610] Lustre: Skipped 1 previous similar message [ 3765.732804] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3767.362690] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3806.177087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3806.179537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3806.183820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3809.253720] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3809.255916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3811.950356] LustreError: 265509:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3811.954039] LustreError: 265509:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3812.020949] Lustre: server umount lustre-OST0000 complete [ 3813.431699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3817.515915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3817.630403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3819.518877] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3821.556758] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3829.925932] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3829.929323] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3835.045191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3835.048995] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3855.842310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.843079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3855.843397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3855.843401] Lustre: Skipped 2 previous similar messages [ 3855.845355] Lustre: Skipped 1 previous similar message [ 3861.164121] LustreError: 267173:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3861.165755] LustreError: 267173:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3861.232131] Lustre: server umount lustre-OST0000 complete [ 3865.568982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3865.571781] LustreError: Skipped 1 previous similar message [ 3865.572882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3868.501629] Lustre: server umount lustre-MDT0000 complete [ 3869.620925] LustreError: 263168:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760141437 with bad export cookie 13855712575806476922 [ 3869.623646] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3869.624025] LustreError: 263168:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3869.681307] LustreError: 267576:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3869.683072] LustreError: 267576:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3869.757376] Lustre: server umount lustre-MDT0001 complete [ 3873.324391] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3874.282815] Key type lgssc unregistered [ 3874.408310] LNet: 268107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3874.410396] LNetError: 268107:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3874.419304] LNet: Removed LNI 192.168.203.145@tcp [ 3874.694787] Key type .llcrypt unregistered [ 3874.695911] Key type ._llcrypt unregistered [ 3883.903952] Key type ._llcrypt registered [ 3883.905110] Key type .llcrypt registered [ 3883.946120] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3884.390750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3884.409248] alg: No test for adler32 (adler32-zlib) [ 3885.288506] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 3885.374616] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3886.952130] Key type lgssc registered [ 3887.289833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3889.920950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3889.930787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3891.118792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3892.619751] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3895.985884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3896.113864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3897.654531] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3899.591540] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3900.685900] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3902.794371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3902.899608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3904.681235] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3906.372910] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3916.200380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3926.498086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3926.499880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3926.500768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3926.501362] LustreError: Skipped 1 previous similar message [ 3926.506155] Lustre: Skipped 1 previous similar message [ 3929.517764] LustreError: 272523:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3929.579147] Lustre: server umount lustre-OST0000 complete [ 3931.618065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3931.622081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3931.623559] Lustre: Skipped 2 previous similar messages [ 3936.738135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3936.741171] Lustre: Skipped 1 previous similar message [ 3936.900397] LustreError: 272725:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3936.903502] LustreError: 272725:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3936.936269] Lustre: server umount lustre-MDT0000 complete [ 3938.095192] LustreError: 272726:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760141505 with bad export cookie 8142306082675516964 [ 3938.097883] LustreError: 272726:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3938.097906] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3938.163719] LustreError: 272927:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3938.165361] LustreError: 272927:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3938.243603] Lustre: server umount lustre-MDT0001 complete [ 3941.766353] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3942.813876] Key type lgssc unregistered [ 3942.946361] LNet: 273458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3942.948333] LNetError: 273458:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3942.956356] LNet: Removed LNI 192.168.203.145@tcp [ 3943.282404] Key type .llcrypt unregistered [ 3943.284031] Key type ._llcrypt unregistered [ 3949.908524] Key type ._llcrypt registered [ 3949.909575] Key type .llcrypt registered [ 3949.958639] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3950.438685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3950.474276] alg: No test for adler32 (adler32-zlib) [ 3951.331189] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 3951.415476] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3952.992166] Key type lgssc registered [ 3953.327797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3955.716082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3955.722388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3956.866438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3958.042581] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3960.712243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3960.833025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3962.112353] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3963.875091] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3965.059699] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3967.230045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3967.358403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3969.099387] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3970.723089] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3981.734832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3991.008868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3991.012156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3991.018129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3992.034688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3992.034893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3992.036441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3995.756867] LustreError: 277050:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3995.812589] Lustre: server umount lustre-OST0000 complete [ 3997.153711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3997.154510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3997.156757] Lustre: Skipped 1 previous similar message [ 3997.158215] Lustre: Skipped 1 previous similar message [ 4002.273094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4002.274860] Lustre: Skipped 2 previous similar messages [ 4003.069376] LustreError: 277252:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4003.071175] LustreError: 277252:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4003.096560] Lustre: server umount lustre-MDT0000 complete [ 4004.278109] LustreError: 274618:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760141572 with bad export cookie 4171346788649785008 [ 4004.280761] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4004.283229] LustreError: 274618:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4004.342997] LustreError: 277453:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4004.344791] LustreError: 277453:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4004.418900] Lustre: server umount lustre-MDT0001 complete [ 4008.081591] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4009.164731] Key type lgssc unregistered [ 4009.302488] LNet: 277985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4009.306228] LNetError: 277985:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4009.314464] LNet: Removed LNI 192.168.203.145@tcp [ 4009.624307] Key type .llcrypt unregistered [ 4009.625297] Key type ._llcrypt unregistered [ 4018.012903] Key type ._llcrypt registered [ 4018.013874] Key type .llcrypt registered [ 4018.053184] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4018.412613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4018.426055] alg: No test for adler32 (adler32-zlib) [ 4019.280067] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 4019.368056] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4020.952097] Key type lgssc registered [ 4021.307231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4023.740933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4023.747409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4024.910461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4026.104328] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4028.801645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4028.933482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4030.120146] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4031.775550] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4032.911823] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4035.066532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4035.185556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4037.013448] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4038.788636] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4049.510718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4062.178362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4062.181034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4062.184958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4064.224776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4064.226748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4064.230416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4065.251183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4065.252925] Lustre: Skipped 1 previous similar message [ 4067.438124] LustreError: 282416:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4067.493850] Lustre: server umount lustre-OST0000 complete [ 4069.856884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4069.860502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4069.865319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4074.750056] LustreError: 282619:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4074.751763] LustreError: 282619:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4074.777858] Lustre: server umount lustre-MDT0000 complete [ 4075.488935] LustreError: 279376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4075.493101] LustreError: 279376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4075.880154] LustreError: 279357:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760141643 with bad export cookie 4829456551403433800 [ 4075.883267] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4075.883272] LustreError: 279357:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4075.946388] LustreError: 282831:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4075.948171] LustreError: 282831:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4076.026048] Lustre: server umount lustre-MDT0001 complete [ 4079.451322] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4080.437605] Key type lgssc unregistered [ 4080.568415] LNet: 283362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4080.570596] LNetError: 283362:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4080.579375] LNet: Removed LNI 192.168.203.145@tcp [ 4080.878921] Key type .llcrypt unregistered [ 4080.879857] Key type ._llcrypt unregistered [ 4087.792059] Key type ._llcrypt registered [ 4087.793026] Key type .llcrypt registered [ 4087.827140] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4088.328844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4088.346513] alg: No test for adler32 (adler32-zlib) [ 4089.201418] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 4089.276126] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4090.848136] Key type lgssc registered [ 4091.146358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4093.408583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4093.414783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4094.575601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4095.697339] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4098.305347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4098.420889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4099.703793] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4101.406595] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4102.510654] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4104.492518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4104.597569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4105.639335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4106.318753] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4107.888448] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4113.891977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4113.893187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4113.893428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4113.896436] Lustre: Skipped 1 previous similar message [ 4118.192994] LustreError: 286970:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4118.261803] Lustre: server umount lustre-OST0000 complete [ 4124.131416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4124.131657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4124.132796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4124.132805] Lustre: Skipped 1 previous similar message [ 4124.140693] Lustre: Skipped 1 previous similar message [ 4125.563151] LustreError: 287171:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4125.564854] LustreError: 287171:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4125.596035] Lustre: server umount lustre-MDT0000 complete [ 4126.809411] LustreError: 284520:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760141694 with bad export cookie 6001608284647182270 [ 4126.815182] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4126.878122] LustreError: 287372:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4126.879857] LustreError: 287372:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4126.969441] Lustre: server umount lustre-MDT0001 complete [ 4130.112771] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4131.261471] Key type lgssc unregistered [ 4131.387515] LNet: 287902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4131.389882] LNetError: 287902:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4131.397394] LNet: Removed LNI 192.168.203.145@tcp [ 4131.739419] Key type .llcrypt unregistered [ 4131.740732] Key type ._llcrypt unregistered [ 4135.463764] Key type ._llcrypt registered [ 4135.464890] Key type .llcrypt registered [ 4135.513915] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4141.170773] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 4146.906799] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4147.257439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4147.266256] alg: No test for adler32 (adler32-zlib) [ 4148.118454] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 4148.198481] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4149.776156] Key type lgssc registered [ 4150.129366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4152.824954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4154.834818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4156.660515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4158.556915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4160.635982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4160.659728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4160.664196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4161.755753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4161.767971] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4161.805137] Lustre: lustre-MDT0000: new disk, initializing [ 4161.830432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4161.836565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4163.039670] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4166.894255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4166.917080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4166.946269] Lustre: 291383:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4166.971463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4166.972985] Lustre: Skipped 1 previous similar message [ 4167.010280] Lustre: lustre-MDT0001: new disk, initializing [ 4167.047604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4167.058464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4167.061601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4168.284382] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4171.219886] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4172.365954] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4174.809419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4174.832704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4174.932123] Lustre: lustre-OST0000: new disk, initializing [ 4174.933828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4174.963942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4177.405656] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4178.200648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4178.206292] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4178.260249] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4181.617241] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4183.662743] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4183.801303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4185.756717] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4185.900159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4188.646796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4188.647930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4188.651618] LustreError: Skipped 1 previous similar message [ 4188.658998] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4188.662824] Lustre: Skipped 1 previous similar message [ 4192.947210] LustreError: 294319:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4193.052091] Lustre: server umount lustre-OST0000 complete [ 4197.857607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4197.862153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4197.869117] Lustre: Skipped 1 previous similar message [ 4197.872375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4198.882164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4198.883200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4201.103141] LustreError: 294520:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4201.107136] LustreError: 294520:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4201.161264] Lustre: server umount lustre-MDT0000 complete [ 4202.804965] LustreError: 292163:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760141770 with bad export cookie 4223731757793764911 [ 4202.811524] LustreError: 292163:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4202.813075] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4202.897345] LustreError: 294721:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4202.900086] LustreError: 294721:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4202.999900] Lustre: server umount lustre-MDT0001 complete [ 4207.228664] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 20:16:13 (1760141773) [ 4210.512416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4210.800480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4212.696278] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4216.289679] LustreError: 295391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4216.293317] LustreError: 295391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4216.662325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4216.839628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4218.340626] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4220.953317] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4222.663836] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4225.978688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4226.160647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4228.630709] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4231.281576] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4240.869334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4240.869455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4240.878434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4240.886613] Lustre: Skipped 2 previous similar messages [ 4245.988590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4245.992055] Lustre: Skipped 1 previous similar message [ 4246.064162] LustreError: 297640:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4246.067780] LustreError: 297640:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4246.159534] Lustre: server umount lustre-OST0000 complete [ 4250.082358] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4250.084531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4250.088018] Lustre: Skipped 1 previous similar message [ 4256.225990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4256.229749] Lustre: Skipped 4 previous similar messages [ 4262.368156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4262.423334] LustreError: 297843:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4262.426968] LustreError: 297843:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4262.462225] Lustre: server umount lustre-MDT0000 complete [ 4264.168347] LustreError: 295370:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760141831 with bad export cookie 4223731757793765779 [ 4264.172539] LustreError: 296112:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4264.175063] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4264.175191] LustreError: 295370:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4264.360498] Lustre: server umount lustre-MDT0001 complete [ 4268.984349] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4270.363438] Key type lgssc unregistered [ 4270.512654] LNet: 298578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4270.517451] LNetError: 298578:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4270.531563] LNet: Removed LNI 192.168.203.145@tcp [ 4270.959729] Key type .llcrypt unregistered [ 4270.962994] Key type ._llcrypt unregistered [ 4282.100301] Key type ._llcrypt registered [ 4282.102158] Key type .llcrypt registered [ 4282.157577] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 4289.644043] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4290.049971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4290.058055] alg: No test for adler32 (adler32-zlib) [ 4290.924054] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 4291.022123] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4292.632169] Key type lgssc registered [ 4293.188061] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4297.245414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4300.645454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4303.507535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4306.613821] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4309.126778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4311.885095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4314.891766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4314.924194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4314.934280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4316.046342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4316.062801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4316.098712] Lustre: lustre-MDT0000: new disk, initializing [ 4316.130681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4316.140706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4317.486516] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4323.038171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4323.078104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4323.139800] Lustre: 302357:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4323.187969] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4323.192048] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4323.195077] Lustre: Skipped 1 previous similar message [ 4323.238339] Lustre: lustre-MDT0001: new disk, initializing [ 4323.286665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4323.306985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4323.312263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4324.652604] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4328.294491] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4330.043578] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4332.852946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4332.884231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4332.994329] Lustre: lustre-OST2710: new disk, initializing [ 4332.998393] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4333.041888] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4335.558885] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4338.193159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4338.199525] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4338.260901] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4339.650904] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4343.009639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4343.047370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4343.135212] Lustre: lustre-OST03e8: new disk, initializing [ 4343.137646] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4343.183951] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4344.489595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4344.495401] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4344.547167] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4345.917634] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4348.641759] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4351.939502] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4352.083400] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4353.858840] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4353.996161] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4355.950890] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4356.104409] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4358.104158] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4358.224428] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4360.165047] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.172193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4360.172770] Lustre: Skipped 3 previous similar messages [ 4360.176426] Lustre: Skipped 2 previous similar messages [ 4365.282130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4365.285878] Lustre: Skipped 2 previous similar messages [ 4365.712300] LustreError: 306702:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4365.749963] Lustre: server umount lustre-MDT0000 complete [ 4367.517950] LustreError: 302347:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760141935 with bad export cookie 16095020044956480189 [ 4367.523423] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4367.525148] LustreError: 302347:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4367.613282] LustreError: 306906:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4367.616847] LustreError: 306906:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4367.690438] Lustre: server umount lustre-MDT0001 complete [ 4379.641171] LustreError: 307106:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4379.645291] LustreError: 307106:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4379.673509] Lustre: server umount lustre-OST2710 complete [ 4391.606698] LustreError: 307308:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4391.610610] LustreError: 307308:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4391.730169] Lustre: server umount lustre-OST03e8 complete [ 4399.678611] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 4404.291315] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4408.813256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4412.123361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4415.199373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4418.449061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4421.754362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4421.792332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4421.922806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4421.938886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4421.991160] Lustre: lustre-MDT0000: new disk, initializing [ 4422.024588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4422.034674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4423.865473] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4429.658701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4429.694418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4429.741104] Lustre: 310101:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4429.860835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4431.335976] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4435.242199] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4437.028286] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4440.478671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4440.518521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4440.678210] Lustre: lustre-OST0000: new disk, initializing [ 4440.679830] Lustre: Skipped 1 previous similar message [ 4440.681919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4440.683832] Lustre: Skipped 2 previous similar messages [ 4440.726581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4440.730429] Lustre: Skipped 1 previous similar message [ 4443.620236] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4444.439121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4444.443883] Lustre: Skipped 1 previous similar message [ 4444.447173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4444.504582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4447.725170] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4449.733314] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4449.847718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4451.735563] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4451.862739] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4454.889477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4454.890907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.894194] LustreError: Skipped 1 previous similar message [ 4454.902338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4454.907437] Lustre: Skipped 1 previous similar message [ 4460.009081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4460.012852] Lustre: Skipped 1 previous similar message [ 4467.168183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4467.194227] LustreError: 313042:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4467.198739] LustreError: 313042:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4467.311439] Lustre: server umount lustre-OST0000 complete [ 4469.218284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4469.219337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4469.225296] Lustre: Skipped 2 previous similar messages [ 4469.231092] Lustre: Skipped 3 previous similar messages [ 4475.344101] LustreError: 313245:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4475.348234] LustreError: 313245:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4475.386413] Lustre: server umount lustre-MDT0000 complete [ 4477.129224] LustreError: 310093:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760142044 with bad export cookie 16095020044956481869 [ 4477.137198] LustreError: 310093:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4477.138231] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4477.320487] Lustre: server umount lustre-MDT0001 complete [ 4481.419460] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 20:20:48 (1760142048) [ 4482.197656] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4482.866974] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 20:20:49 (1760142049) [ 4483.664084] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4487.092434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4488.887057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4490.588778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4492.568319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4493.111078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4496.269140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4496.415285] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4496.429099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4496.513630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4498.183576] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4501.808452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4501.837604] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4501.846265] Lustre: 315572:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4503.324965] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4505.275442] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4507.032235] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4510.306198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4510.347885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4510.445977] Lustre: Denying initial registration attempt from nid 192.168.203.145@tcp, specified as failover [ 4510.451199] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4510.455036] LustreError: 317189:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 4510.457864] LustreError: 317189:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4510.460409] LustreError: 317189:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4510.522815] Lustre: server umount lustre-OST0000 complete [ 4510.524608] LustreError: 317189:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4517.347326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4517.347495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4517.348283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4517.348292] Lustre: Skipped 2 previous similar messages [ 4517.350502] Lustre: Skipped 1 previous similar message [ 4519.577275] LustreError: 317557:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4519.581204] LustreError: 317557:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4519.622946] Lustre: server umount lustre-MDT0000 complete [ 4521.460323] LustreError: 315563:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760142089 with bad export cookie 16095020044956482751 [ 4521.466445] LustreError: 315563:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4521.468487] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4521.687879] Lustre: server umount lustre-MDT0001 complete [ 4526.167106] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4527.731140] Key type lgssc unregistered [ 4527.909837] LNet: 318293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4527.914466] LNetError: 318293:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4527.923846] LNet: Removed LNI 192.168.203.145@tcp [ 4528.440846] Key type .llcrypt unregistered [ 4528.443089] Key type ._llcrypt unregistered [ 4532.666329] Key type ._llcrypt registered [ 4532.668554] Key type .llcrypt registered [ 4532.714501] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4533.199416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4533.229407] alg: No test for adler32 (adler32-zlib) [ 4534.092188] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 4534.178284] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4535.768278] Key type lgssc registered [ 4536.407384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4539.870427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4541.686621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4543.420452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4544.610903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4545.007587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4553.896579] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4557.907764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4557.922932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4559.069910] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4559.082230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4559.164492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4560.982184] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4565.363755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4565.420159] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4565.441822] Lustre: 321024:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4565.645606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4567.539804] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4570.051858] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4571.674860] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4574.563880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4574.602092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4574.708207] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4574.820087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4577.268498] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4580.093823] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4581.195191] LustreError: 323300:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4581.294066] Lustre: server umount lustre-OST0000 complete [ 4586.464924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4586.469048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.475375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4587.490644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4587.492312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4589.070228] LustreError: 323509:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 4589.074077] LustreError: 323509:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4589.110137] Lustre: server umount lustre-MDT0000 complete [ 4590.853336] LustreError: 323510:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760142158 with bad export cookie 2110843845149642777 [ 4590.859422] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4590.860019] LustreError: 323510:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4590.948169] LustreError: 323711:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4590.951199] LustreError: 323711:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4591.034793] Lustre: server umount lustre-MDT0001 complete [ 4596.403334] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4598.037783] Key type lgssc unregistered [ 4598.169414] LNet: 324242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4598.171857] LNetError: 324242:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4598.179565] LNet: Removed LNI 192.168.203.145@tcp [ 4598.588232] Key type .llcrypt unregistered [ 4598.590328] Key type ._llcrypt unregistered [ 4609.824443] Key type ._llcrypt registered [ 4609.825576] Key type .llcrypt registered [ 4609.862960] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4610.169756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4610.182698] alg: No test for adler32 (adler32-zlib) [ 4611.103402] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 4611.238299] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4612.848223] Key type lgssc registered [ 4613.498846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4617.084714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4617.097677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4618.336881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4620.060728] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4623.738636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4623.922609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4625.493134] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4628.133642] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4629.965382] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4633.269373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4633.449594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4635.790783] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4638.228259] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4654.008653] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 4654.561643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4654.566260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4654.574187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4655.076179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4655.083952] Lustre: Skipped 1 previous similar message [ 4657.596517] LustreError: 327890:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4657.623273] Lustre: server umount lustre-MDT0000 complete [ 4658.931849] LustreError: 325641:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760142226 with bad export cookie 1316427173358518701 [ 4658.937119] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4658.938812] LustreError: 325641:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4659.115174] LustreError: 327235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4659.125696] LustreError: 327235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4659.126543] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 4659.131349] Lustre: Skipped 3 previous similar messages [ 4660.193712] LustreError: 326826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4660.194110] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4663.200663] LustreError: 326826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4663.200876] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4663.206206] Lustre: Skipped 1 previous similar message [ 4665.090402] LustreError: 328092:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4665.092686] LustreError: 328092:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4665.169254] Lustre: server umount lustre-MDT0001 complete [ 4668.020981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4671.527611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4683.744634] LustreError: 324926:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e130169e x1845642845511296/t0(0) o250->MGC192.168.203.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4683.845074] LustreError: 327226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4683.850025] LustreError: 327226:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4683.866900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4683.886098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4685.312016] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4688.810206] LustreError: 328902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4688.812256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4688.865694] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4688.867487] Lustre: Skipped 1 previous similar message [ 4688.931551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4689.960771] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4690.605287] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4692.920314] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4694.053381] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4695.808603] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4697.291507] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4698.076254] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4704.231422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4704.234601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4704.240828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4704.243367] LustreError: Skipped 1 previous similar message [ 4704.246984] Lustre: Skipped 1 previous similar message [ 4705.395210] LustreError: 330394:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4705.399532] LustreError: 330394:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4705.505272] Lustre: server umount lustre-OST0000 complete [ 4709.348918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4709.349780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4709.362276] Lustre: Skipped 1 previous similar message [ 4713.357725] LustreError: 330597:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4713.362463] LustreError: 330597:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4713.407779] Lustre: server umount lustre-MDT0000 complete [ 4714.466600] LustreError: 328906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4714.475751] LustreError: 328906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4715.259581] LustreError: 325641:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760142283 with bad export cookie 1316427173358525855 [ 4715.265963] LustreError: 325641:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4715.267344] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4715.494067] Lustre: server umount lustre-MDT0001 complete [ 4720.049362] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4721.502618] Key type lgssc unregistered [ 4721.658537] LNet: 331329:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4721.661922] LNetError: 331329:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4721.671782] LNet: Removed LNI 192.168.203.145@tcp [ 4722.097490] Key type .llcrypt unregistered [ 4722.099747] Key type ._llcrypt unregistered [ 4733.573281] Key type ._llcrypt registered [ 4733.575577] Key type .llcrypt registered [ 4733.640585] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4734.198992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4734.209115] alg: No test for adler32 (adler32-zlib) [ 4735.073657] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 4735.168516] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4736.744193] Key type lgssc registered [ 4737.322858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4741.093749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4741.105909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4742.341829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4744.261135] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4748.484072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4748.679397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4750.510817] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4753.133203] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4754.814351] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4757.989960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4757.992752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4757.997061] Lustre: Skipped 1 previous similar message [ 4758.003294] Lustre: Skipped 1 previous similar message [ 4759.011491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4759.015161] Lustre: Skipped 1 previous similar message [ 4762.507143] LustreError: 334094:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4762.546184] Lustre: server umount lustre-MDT0000 complete [ 4764.130764] LustreError: 333427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4764.138916] LustreError: 333427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4764.155284] LustreError: 333421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760142331 with bad export cookie 18295036661371444588 [ 4764.160665] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4764.160825] LustreError: 333421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4764.244879] LustreError: 334299:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4764.246884] LustreError: 334299:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4764.356413] Lustre: server umount lustre-MDT0001 complete [ 4771.507810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4771.650102] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4771.663611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4771.761964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4773.699061] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4778.000368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4778.061652] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4778.082426] Lustre: 335068:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4778.245349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4779.978697] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4782.531612] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4784.240688] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4787.737786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4787.847843] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4787.853883] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4787.858335] LustreError: 336728:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4787.861893] LustreError: 336728:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4787.864881] LustreError: 336728:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4787.934594] Lustre: server umount lustre-OST0000 complete [ 4787.936237] LustreError: 336728:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4791.907161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4792.024396] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4792.141319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4794.781236] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4797.339623] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4798.525920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4801.138839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4801.178540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4801.215351] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4801.284941] Lustre: lustre-OST0001: new disk, initializing [ 4801.288391] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4801.345779] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4804.005434] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4806.744094] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4807.729644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4807.735452] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4807.793353] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4812.776339] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4812.780601] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4812.787116] Lustre: Skipped 1 previous similar message [ 4812.787206] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4812.792984] Lustre: Skipped 1 previous similar message [ 4814.330625] LustreError: 338847:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4814.334075] LustreError: 338847:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4814.365382] Lustre: server umount lustre-OST0001 complete [ 4817.897222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4817.900862] LustreError: 337195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4817.901832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4817.903124] Lustre: Skipped 1 previous similar message [ 4817.910590] LustreError: 337195:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4822.320190] LustreError: 339048:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4822.324849] LustreError: 339048:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4822.427244] Lustre: server umount lustre-OST0000 complete [ 4824.545628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4824.552608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4824.563363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4824.567405] Lustre: Skipped 1 previous similar message [ 4830.481729] LustreError: 339250:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4830.484754] LustreError: 339250:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4830.510800] Lustre: server umount lustre-MDT0000 complete [ 4832.287116] LustreError: 335790:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760142400 with bad export cookie 18295036661371445127 [ 4832.294055] LustreError: 335790:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4832.294269] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4832.495137] Lustre: server umount lustre-MDT0001 complete [ 4837.547456] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4839.076902] Key type lgssc unregistered [ 4839.230720] LNet: 339983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4839.235403] LNetError: 339983:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4839.245608] LNet: Removed LNI 192.168.203.145@tcp [ 4839.719845] Key type .llcrypt unregistered [ 4839.722074] Key type ._llcrypt unregistered [ 4854.831850] Key type ._llcrypt registered [ 4854.833966] Key type .llcrypt registered [ 4854.894741] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 4863.333752] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4863.640469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4863.678822] alg: No test for adler32 (adler32-zlib) [ 4864.609249] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 4864.732541] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4866.344186] Key type lgssc registered [ 4866.938170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4871.603356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4875.052340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4878.143307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4881.457691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4884.566065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4884.598525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4884.606144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4885.725229] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4885.746722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4885.791827] Lustre: lustre-MDT0000: new disk, initializing [ 4885.832844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4885.845348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4887.776866] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4893.545411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4893.585765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4893.645727] Lustre: 343905:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4893.693459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4893.696967] Lustre: Skipped 1 previous similar message [ 4893.747851] Lustre: lustre-MDT0001: new disk, initializing [ 4893.809360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4893.830767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4893.837645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4895.773800] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4900.010285] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4901.642283] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4904.991557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4905.033095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4905.184630] Lustre: lustre-OST0000: new disk, initializing [ 4905.187381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4905.238291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4907.911581] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4911.114147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4911.120157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4911.173107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4911.896396] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4913.660607] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4913.747667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4915.241139] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4915.316190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4921.318253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4921.318492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4921.323717] LustreError: Skipped 1 previous similar message [ 4921.331654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4921.337642] Lustre: Skipped 1 previous similar message [ 4922.290135] LustreError: 346846:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4922.390174] Lustre: server umount lustre-OST0000 complete [ 4924.385686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4924.390152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4924.397146] Lustre: Skipped 1 previous similar message [ 4924.401674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4926.434514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4926.436423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4930.318570] LustreError: 347047:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4930.322454] LustreError: 347047:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4930.373620] Lustre: server umount lustre-MDT0000 complete [ 4932.232350] LustreError: 347049:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760142500 with bad export cookie 11516089503585867049 [ 4932.241070] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4932.243256] LustreError: 347049:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4932.333833] LustreError: 347250:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4932.337879] LustreError: 347250:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4932.458061] Lustre: server umount lustre-MDT0001 complete [ 4936.635835] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 20:28:23 (1760142503) [ 4939.521202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4942.442958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4952.494497] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 4957.187792] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4962.024147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4965.463800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4968.628393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4971.863170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4975.212652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4975.249657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4975.404488] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4975.422375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4975.477441] Lustre: lustre-MDT0000: new disk, initializing [ 4975.515135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4975.526326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4977.347531] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4982.738990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4982.770559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4982.827577] Lustre: 351022:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4982.859383] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4982.862167] Lustre: Skipped 1 previous similar message [ 4982.903575] Lustre: lustre-MDT0001: new disk, initializing [ 4982.940098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4982.954178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4982.958376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4984.540314] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4988.640808] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4990.473412] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4993.917362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4993.962924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4994.129981] Lustre: lustre-OST0000: new disk, initializing [ 4994.133159] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4994.186784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4996.977487] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5000.942467] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5002.733946] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5003.272929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5003.278689] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5003.332178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5003.882849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5005.657536] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5005.763806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5008.358847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5008.359067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5008.372109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5008.375713] Lustre: Skipped 2 previous similar messages [ 5012.720171] LustreError: 353968:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5012.724048] LustreError: 353968:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5012.827552] Lustre: server umount lustre-OST0000 complete [ 5018.594432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5018.595362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5018.600402] Lustre: Skipped 2 previous similar messages [ 5020.684580] LustreError: 354169:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5020.688473] LustreError: 354169:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5020.725251] Lustre: server umount lustre-MDT0000 complete [ 5022.363860] LustreError: 351012:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760142590 with bad export cookie 11516089503585867924 [ 5022.369701] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5022.372877] LustreError: 351012:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5022.555831] Lustre: server umount lustre-MDT0001 complete [ 5026.359200] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 20:29:53 (1760142593) [ 5029.974639] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 20:29:56 (1760142596) [ 5033.942950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5034.167922] LustreError: 355386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5034.173324] LustreError: 355386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5034.188771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5035.829579] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5039.586216] LustreError: 355386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5040.011303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5042.040327] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5044.777438] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5046.593481] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5049.913867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5052.741344] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5055.350348] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5069.781257] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5070.678345] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5071.476431] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5072.405055] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5073.400745] Lustre: Failing over lustre-MDT0000 [ 5073.600550] LustreError: 357846:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5073.603383] LustreError: 357846:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5073.636444] Lustre: server umount lustre-MDT0000 complete [ 5074.917461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5074.919712] LustreError: 357537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5074.925829] Lustre: Skipped 2 previous similar messages [ 5074.939166] LustreError: 357537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5077.431104] LustreError: 357537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5082.540167] LustreError: 355382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5082.550369] LustreError: 355382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5087.859980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5087.928647] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5088.071269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5088.076579] Lustre: Skipped 2 previous similar messages [ 5088.119769] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5089.871301] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5092.781447] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5093.354597] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5093.366639] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5093.397789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5095.182573] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5096.045233] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5098.407091] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5099.317627] Lustre: Failing over lustre-MDT0000 [ 5099.463227] LustreError: 358925:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5099.465584] LustreError: 358925:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5099.491949] Lustre: server umount lustre-MDT0000 complete [ 5103.033450] LustreError: 355381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5103.043197] LustreError: 355381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5103.589149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5103.589619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.593805] LustreError: Skipped 1 previous similar message [ 5113.679652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5113.753972] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5113.825700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 5113.828195] Lustre: Skipped 3 previous similar messages [ 5113.958258] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5115.677781] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5116.362558] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5118.959894] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5118.963675] Lustre: Skipped 2 previous similar messages [ 5118.972833] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5119.001512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5120.764650] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5121.580654] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5129.189080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5130.543306] Lustre: server umount lustre-MDT0000 complete [ 5132.330177] LustreError: 355366:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760142700 with bad export cookie 11516089503585870563 [ 5132.335987] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5132.337057] LustreError: 355366:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5132.428684] LustreError: 360155:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5132.431550] LustreError: 360155:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5132.518162] Lustre: server umount lustre-MDT0001 complete [ 5144.730379] Lustre: server umount lustre-OST0000 complete [ 5152.162988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5152.398634] LustreError: 361183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5152.409208] LustreError: 361183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5152.441212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5152.444981] Lustre: Skipped 1 previous similar message [ 5154.291717] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5158.571514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5160.718546] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5163.468215] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5165.173296] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5168.330120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5171.240158] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5173.908907] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5177.773678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5182.950560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5182.950734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5182.957551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5182.965646] Lustre: Skipped 6 previous similar messages [ 5182.972445] Lustre: Skipped 4 previous similar messages [ 5186.859912] Lustre: server umount lustre-OST0000 complete [ 5194.541489] Lustre: server umount lustre-MDT0000 complete [ 5196.343887] LustreError: 362801:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760142764 with bad export cookie 11516089503585871291 [ 5196.351766] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5196.353420] LustreError: 362801:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5196.446106] LustreError: 363834:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5196.449837] LustreError: 363834:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 5196.537117] Lustre: server umount lustre-MDT0001 complete [ 5201.516115] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5202.976654] Key type lgssc unregistered [ 5203.145654] LNet: 364365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5203.150420] LNetError: 364365:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5203.161528] LNet: Removed LNI 192.168.203.145@tcp [ 5203.617931] Key type .llcrypt unregistered [ 5203.620142] Key type ._llcrypt unregistered [ 5216.890302] Key type ._llcrypt registered [ 5216.892129] Key type .llcrypt registered [ 5216.952858] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5217.368117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5217.430092] alg: No test for adler32 (adler32-zlib) [ 5218.336997] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 5218.443096] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 5220.048195] Key type lgssc registered [ 5220.679338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5224.664291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5224.676972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5225.917933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5227.923626] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5232.210087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5232.435115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5234.381976] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5237.081658] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5238.873738] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5242.259697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5242.467747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5245.155804] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5247.772669] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5250.675631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5258.209349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5258.213873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5258.221851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5261.283159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5261.288864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5263.631786] LustreError: 368085:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5263.666604] Lustre: server umount lustre-MDT0000 complete [ 5265.432970] LustreError: 365821:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760142833 with bad export cookie 13604153064017448325 [ 5265.438539] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5265.439460] LustreError: 365821:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5265.527502] LustreError: 368288:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5265.530111] LustreError: 368288:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5265.614458] Lustre: server umount lustre-MDT0001 complete [ 5277.621960] LustreError: 368489:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5277.625809] LustreError: 368489:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5277.728341] Lustre: server umount lustre-OST0000 complete [ 5284.279738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5284.503730] LustreError: 369220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5284.514683] LustreError: 369220:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5284.543797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5286.261310] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5289.955623] LustreError: 369220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5290.434323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5290.608703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5292.496278] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5295.065922] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5296.891430] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5300.304139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5300.497238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5303.260512] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5305.895318] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5309.748902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5320.167048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5320.170344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5320.174936] Lustre: Skipped 3 previous similar messages [ 5320.181178] Lustre: Skipped 4 previous similar messages [ 5324.942810] LustreError: 371570:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5324.946321] LustreError: 371570:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5324.987093] Lustre: server umount lustre-MDT0000 complete [ 5325.285972] LustreError: 370336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5325.298451] LustreError: 370336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5326.870428] LustreError: 371572:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760142894 with bad export cookie 13604153064017449438 [ 5326.878365] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5326.881427] LustreError: 371572:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5327.099837] Lustre: server umount lustre-MDT0001 complete [ 5338.230742] LustreError: 371973:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5338.234477] LustreError: 371973:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5338.331223] Lustre: server umount lustre-OST0000 complete [ 5345.670394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5345.890903] LustreError: 372801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5345.930614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5347.659616] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5351.393730] LustreError: 372801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5351.713984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5353.756180] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5356.261477] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5358.032064] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5361.341577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5364.112282] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5366.799317] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5370.733504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5375.975763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5375.975826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5375.983856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5375.990800] Lustre: Skipped 1 previous similar message [ 5379.826141] LustreError: 375047:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5379.829947] LustreError: 375047:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5379.924520] Lustre: server umount lustre-OST0000 complete [ 5382.625377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5382.629884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5382.636902] Lustre: Skipped 1 previous similar message [ 5382.640260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5387.884555] Lustre: server umount lustre-MDT0000 complete [ 5389.655311] LustreError: 372780:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760142957 with bad export cookie 13604153064017450460 [ 5389.662391] LustreError: 372780:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5389.662610] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5389.844816] Lustre: server umount lustre-MDT0001 complete [ 5394.607375] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5396.089462] Key type lgssc unregistered [ 5396.221837] LNet: 375984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5396.226452] LNetError: 375984:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5396.238594] LNet: Removed LNI 192.168.203.145@tcp [ 5396.714270] Key type .llcrypt unregistered [ 5396.716373] Key type ._llcrypt unregistered [ 5409.398582] Key type ._llcrypt registered [ 5409.400298] Key type .llcrypt registered [ 5409.451914] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5409.764173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5409.774227] alg: No test for adler32 (adler32-zlib) [ 5410.646154] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 5410.752670] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 5412.352235] Key type lgssc registered [ 5412.933893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5416.855535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5416.866346] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5416.871772] LustreError: 377382:0:(osd_handler.c:8486:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5416.884150] LustreError: 377382:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5416.888463] LustreError: 377382:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5416.892830] LustreError: 377382:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5416.897790] LustreError: 377382:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5420.608258] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5420.612391] LustreError: 377779:0:(osd_handler.c:8486:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5420.624375] LustreError: 377779:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5420.628063] LustreError: 377779:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5420.632091] LustreError: 377779:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5420.636994] LustreError: 377779:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5429.934376] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5431.538341] Key type lgssc unregistered [ 5431.728580] LNet: 378764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5431.732282] LNetError: 378764:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5431.743623] LNet: Removed LNI 192.168.203.145@tcp [ 5432.242897] Key type .llcrypt unregistered [ 5432.245145] Key type ._llcrypt unregistered [ 5440.724412] Key type ._llcrypt registered [ 5440.726521] Key type .llcrypt registered [ 5440.797596] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 5448.649243] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5449.137826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5449.144939] alg: No test for adler32 (adler32-zlib) [ 5450.027310] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 5450.140514] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 5451.752151] Key type lgssc registered [ 5452.349799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5456.772351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5459.985953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5463.063468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5466.355658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5469.804830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5469.834262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5469.846836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5470.985324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5471.006372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5471.066250] Lustre: lustre-MDT0000: new disk, initializing [ 5471.110602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5471.123816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5473.047404] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5478.718531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5478.757136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5478.813381] Lustre: 381869:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5478.857911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5478.861195] Lustre: Skipped 1 previous similar message [ 5478.904548] Lustre: lustre-MDT0001: new disk, initializing [ 5478.950396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5478.972093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5478.977439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5480.892148] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5485.110779] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5487.007328] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5490.450383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5490.498626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5490.654731] Lustre: lustre-OST0000: new disk, initializing [ 5490.657556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5490.703644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5493.513614] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5496.840875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5496.848321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5496.908224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5497.679696] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5499.735379] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5499.870029] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5501.796365] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5501.932088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5507.050486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5507.051821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5507.053962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5507.054989] LustreError: Skipped 1 previous similar message [ 5507.068400] Lustre: Skipped 1 previous similar message [ 5512.169848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5512.173587] Lustre: Skipped 2 previous similar messages [ 5517.280263] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5517.289887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5517.293496] Lustre: Skipped 1 previous similar message [ 5517.317969] LustreError: 384808:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5517.421181] Lustre: server umount lustre-OST0000 complete [ 5519.841622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5519.846232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5519.855894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5522.404774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5525.389686] LustreError: 385010:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5525.393401] LustreError: 385010:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5525.437322] Lustre: server umount lustre-MDT0000 complete [ 5527.229313] LustreError: 381860:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760143095 with bad export cookie 9989691684421510365 [ 5527.238140] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5527.239294] LustreError: 381860:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5527.334238] LustreError: 385214:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5527.337761] LustreError: 385214:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5527.452304] Lustre: server umount lustre-MDT0001 complete [ 5531.281396] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5532.120697] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 20:38:18 (1760143098) [ 5535.358182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5535.659227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5537.424213] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5540.835060] LustreError: 385932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5540.844320] LustreError: 385932:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5541.453057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5541.640238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5543.230371] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5545.785201] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5547.583077] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5550.832819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5550.990437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5553.630201] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5556.341155] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5559.650094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5559.687981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5559.775725] Lustre: lustre-OST0001: new disk, initializing [ 5559.778978] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5562.395148] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5566.547379] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5569.544837] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5569.548478] Lustre: Skipped 2 previous similar messages [ 5574.832221] LustreError: 389120:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5574.836110] LustreError: 389120:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5574.862751] Lustre: server umount lustre-OST0001 complete [ 5579.751432] LustreError: 387529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5579.760577] LustreError: 387529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5584.874342] LustreError: 387529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5584.883383] LustreError: 387529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5587.938705] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5587.943261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5587.951207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5587.954855] Lustre: Skipped 4 previous similar messages [ 5589.995267] LustreError: 389127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5590.003849] LustreError: 389127:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5593.714232] LustreError: 389353:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5593.718300] LustreError: 389353:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5593.818288] Lustre: server umount lustre-OST0000 complete [ 5598.177797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5598.182310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5598.189363] Lustre: Skipped 1 previous similar message [ 5601.873375] LustreError: 389555:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5601.877257] LustreError: 389555:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5601.923437] Lustre: server umount lustre-MDT0000 complete [ 5603.744163] LustreError: 387551:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760143171 with bad export cookie 9989691684421511247 [ 5603.751131] LustreError: 387551:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5603.752648] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5618.144146] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5618.201927] LustreError: 389758:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5618.205815] LustreError: 389758:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5618.329044] Lustre: server umount lustre-MDT0001 complete [ 5623.390294] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5624.962378] Key type lgssc unregistered [ 5625.129538] LNet: 390291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5625.134215] LNetError: 390291:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5625.144650] LNet: Removed LNI 192.168.203.145@tcp [ 5625.632153] Key type .llcrypt unregistered [ 5625.634288] Key type ._llcrypt unregistered [ 5641.670808] Key type ._llcrypt registered [ 5641.672174] Key type .llcrypt registered [ 5641.725310] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 5649.957286] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5650.507301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5650.519720] alg: No test for adler32 (adler32-zlib) [ 5651.452167] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 5651.580418] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 5653.192230] Key type lgssc registered [ 5653.827944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5658.496729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5661.731653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5664.720947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5667.874421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5671.250072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5671.285126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5671.295630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5672.413304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5672.433663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5672.478100] Lustre: lustre-MDT0000: new disk, initializing [ 5672.516782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5672.529284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5674.385632] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5679.952740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5679.989183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5680.046587] Lustre: 394215:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5680.089181] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5680.092722] Lustre: Skipped 1 previous similar message [ 5680.139189] Lustre: lustre-MDT0001: new disk, initializing [ 5680.198524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5680.221365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5680.228105] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5682.126542] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5686.335512] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5688.228669] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5691.512678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5691.561619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5691.713305] Lustre: lustre-OST0000: new disk, initializing [ 5691.716453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5691.763572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5694.205415] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5695.114740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5695.121143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5695.177463] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5698.244826] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5699.920065] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5700.045170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5701.730043] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5701.815895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5705.703540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.711497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5705.711863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5705.719427] Lustre: Skipped 1 previous similar message [ 5710.823052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5710.826725] Lustre: Skipped 1 previous similar message [ 5715.942844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5715.946512] Lustre: Skipped 1 previous similar message [ 5716.960163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5716.984654] LustreError: 397154:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5717.076739] Lustre: server umount lustre-OST0000 complete [ 5721.060071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5721.060369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5721.061276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5721.064649] LustreError: Skipped 1 previous similar message [ 5721.071846] Lustre: Skipped 2 previous similar messages [ 5724.748092] LustreError: 397358:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5724.751988] LustreError: 397358:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5724.794689] Lustre: server umount lustre-MDT0000 complete [ 5726.580132] LustreError: 397359:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760143294 with bad export cookie 12883219682561366358 [ 5726.585844] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5726.587060] LustreError: 397359:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5726.681833] LustreError: 397560:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5726.685446] LustreError: 397560:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5726.783680] Lustre: server umount lustre-MDT0001 complete [ 5730.835982] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 20:41:37 (1760143297) [ 5735.079116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5739.093910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.314646] Lustre: lustre-MDT0000: new disk, initializing [ 5739.332358] LustreError: 398825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5739.341628] LustreError: 398825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5739.367554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5741.106196] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5744.610529] LustreError: 398825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.348044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5745.537664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5747.332697] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5749.981987] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5751.770749] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5754.850928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5754.851801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5754.858579] Lustre: Skipped 1 previous similar message [ 5754.862115] Lustre: Skipped 1 previous similar message [ 5758.860435] LustreError: 400166:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5758.864479] LustreError: 400166:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5758.900302] Lustre: server umount lustre-MDT0000 complete [ 5760.686880] LustreError: 398806:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760143328 with bad export cookie 12883219682561367240 [ 5760.692218] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5760.693434] LustreError: 398806:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5760.867688] Lustre: server umount lustre-MDT0001 complete [ 5765.276667] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 20:42:12 (1760143332) [ 5768.578655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5768.830943] LustreError: 401088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5768.877798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5770.714738] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5774.306264] LustreError: 401088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5774.835603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5776.884907] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5779.092857] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5780.159268] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5783.329693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5786.028688] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5788.661358] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5796.653524] Lustre: Permanently deactivating lustre-OST0000 [ 5796.658272] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5806.065392] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5807.436238] LustreError: 403923:0:(mgs_llog.c:1937:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5807.439043] LustreError: 403923:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5807.440883] LustreError: 403923:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5808.006328] LustreError: 403971:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5808.008273] LustreError: 403971:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5814.638732] LustreError: 404068:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5814.640435] LustreError: 404068:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5814.694561] Lustre: server umount lustre-OST0000 complete [ 5815.509821] LustreError: 404221:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5815.511838] LustreError: 404221:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5816.288741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5816.290853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5816.294326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5816.295853] Lustre: Skipped 3 previous similar messages [ 5820.386624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5825.510083] LustreError: 404318:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5825.511767] LustreError: 404318:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5825.532688] Lustre: server umount lustre-MDT0000 complete [ 5826.668564] LustreError: 401068:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760143394 with bad export cookie 12883219682561367779 [ 5826.671403] LustreError: 401068:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5826.671646] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5836.334955] Lustre: server umount lustre-MDT0001 complete [ 5838.984653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5840.231878] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5841.090094] LustreError: 405363:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5843.018102] Lustre: 405701:0:(mgs_llog.c:1698:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.145@tcp [ 5843.649441] LustreError: 405798:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5843.651130] LustreError: 405798:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5843.674842] Lustre: server umount MGS complete [ 5847.179190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5847.311348] LustreError: 406426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5847.315423] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5847.317527] Lustre: Skipped 1 previous similar message [ 5847.334030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5847.335605] Lustre: Skipped 2 previous similar messages [ 5848.513536] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5851.138857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5851.225265] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5852.562818] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5855.068388] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5856.877450] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5857.782949] Lustre: Permanently reactivating lustre-OST0000 [ 5857.792512] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5857.796710] Lustre: Skipped 2 previous similar messages [ 5886.944265] LustreError: 408093:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5890.610673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5890.727858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5890.731278] Lustre: Skipped 1 previous similar message [ 5892.970603] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5895.664308] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5912.549912] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5912.552582] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5912.553550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5912.553560] Lustre: Skipped 4 previous similar messages [ 5912.556878] Lustre: Skipped 1 previous similar message [ 5914.219109] LustreError: 409856:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5914.221077] LustreError: 409856:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5914.306267] Lustre: server umount lustre-OST0000 complete [ 5917.665509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5917.667183] LustreError: Skipped 1 previous similar message [ 5921.865266] Lustre: server umount lustre-MDT0000 complete [ 5922.784932] LustreError: 406426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5922.788942] LustreError: 406426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5922.875071] LustreError: 409757:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760143490 with bad export cookie 12883219682561368724 [ 5922.877689] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5922.878147] LustreError: 409757:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5923.003226] Lustre: server umount lustre-MDT0001 complete [ 5926.094192] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5926.992738] Key type lgssc unregistered [ 5927.114277] LNet: 410794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5927.116212] LNetError: 410794:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5927.125315] LNet: Removed LNI 192.168.203.145@tcp [ 5927.379830] Key type .llcrypt unregistered [ 5927.380783] Key type ._llcrypt unregistered [ 5932.655542] Key type ._llcrypt registered [ 5932.656510] Key type .llcrypt registered [ 5932.692229] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 5938.321923] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5938.577884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5938.607228] alg: No test for adler32 (adler32-zlib) [ 5939.474581] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 5939.584015] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 5941.176100] Key type lgssc registered [ 5941.662609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5944.255624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5946.192394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5947.992350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5949.801826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5951.816802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5951.835997] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5951.840712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5952.917013] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5952.926187] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5952.959969] Lustre: lustre-MDT0000: new disk, initializing [ 5952.981454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5952.986920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5954.099033] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5957.848324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5957.873470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5957.900341] Lustre: 413898:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5957.924267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5957.925663] Lustre: Skipped 1 previous similar message [ 5957.958911] Lustre: lustre-MDT0001: new disk, initializing [ 5958.001792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5958.014886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5958.017740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5959.141249] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5961.954530] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5962.976497] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5964.999481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5965.026933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5965.117473] Lustre: lustre-OST0000: new disk, initializing [ 5965.119187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5965.148988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5966.780868] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5969.586262] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5970.734671] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5971.451058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5971.454156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5971.490823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5971.823944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5972.937407] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5973.014168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5976.548606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5976.550236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5976.550866] LustreError: Skipped 1 previous similar message [ 5976.554946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5976.556961] Lustre: Skipped 1 previous similar message [ 5979.629293] LustreError: 416843:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5979.680296] Lustre: server umount lustre-OST0000 complete [ 5981.666087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5981.667127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5981.668881] Lustre: Skipped 2 previous similar messages [ 5986.786878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5986.789089] Lustre: Skipped 2 previous similar messages [ 5991.905154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5991.906688] Lustre: Skipped 1 previous similar message [ 5994.976172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5995.007032] LustreError: 417044:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5995.008753] LustreError: 417044:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5995.037895] Lustre: server umount lustre-MDT0000 complete [ 5996.111205] LustreError: 414678:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760143563 with bad export cookie 18241512144666303225 [ 5996.115324] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5996.176833] LustreError: 417247:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5996.178438] LustreError: 417247:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5996.254336] Lustre: server umount lustre-MDT0001 complete [ 5998.619742] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5999.086794] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:46:06 (1760143566) [ 6001.067704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6001.221626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6002.307106] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6004.721246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6005.932596] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6007.424685] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6008.548330] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6011.761981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6011.875866] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 6011.911439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6011.913855] Lustre: Skipped 1 previous similar message [ 6014.523561] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6017.078113] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6017.998127] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6019.681052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6025.186407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6025.187793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6025.189647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6025.190308] LustreError: Skipped 1 previous similar message [ 6025.193160] Lustre: Skipped 1 previous similar message [ 6027.309121] LustreError: 420404:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6027.311197] LustreError: 420404:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6027.376042] Lustre: server umount lustre-OST0000 complete [ 6030.307434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6030.307436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6030.314688] Lustre: Skipped 1 previous similar message [ 6034.622705] LustreError: 420606:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6034.625881] LustreError: 420606:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6034.650983] Lustre: server umount lustre-MDT0000 complete [ 6035.424960] LustreError: 417966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6035.428983] LustreError: 417966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6035.772577] LustreError: 417946:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760143603 with bad export cookie 18241512144666304107 [ 6035.775706] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6035.776801] LustreError: 417946:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6035.904241] Lustre: server umount lustre-MDT0001 complete [ 6039.169278] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6040.179125] Key type lgssc unregistered [ 6040.307389] LNet: 421339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6040.309385] LNetError: 421339:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6040.320388] LNet: Removed LNI 192.168.203.145@tcp [ 6040.627105] Key type .llcrypt unregistered [ 6040.628131] Key type ._llcrypt unregistered [ 6047.360672] Key type ._llcrypt registered [ 6047.361673] Key type .llcrypt registered [ 6047.398402] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6047.496247] Key type .llcrypt unregistered [ 6047.497179] Key type ._llcrypt unregistered [ 6053.847801] Key type ._llcrypt registered [ 6053.848739] Key type .llcrypt registered [ 6053.886036] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6054.262107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6054.327354] alg: No test for adler32 (adler32-zlib) [ 6055.183973] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 6055.264324] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 6056.840112] Key type lgssc registered [ 6057.201134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6059.685126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6059.690909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6060.856989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6062.101645] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6064.720204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6064.839660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6066.571377] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6068.129940] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6070.228871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6070.245868] LustreError: 423485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6070.251551] LustreError: 423485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6070.358386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6071.405653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6071.570947] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6075.360951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6075.363287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6075.367505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6076.903675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.909497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6076.911787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6076.913634] Lustre: Skipped 1 previous similar message [ 6080.301128] LustreError: 425539:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6080.359354] Lustre: server umount lustre-OST0000 complete [ 6082.018731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6082.019449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6082.023513] Lustre: Skipped 1 previous similar message [ 6087.138581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6087.142265] Lustre: Skipped 2 previous similar messages [ 6087.629053] LustreError: 425741:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6087.632864] LustreError: 425741:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6087.677469] Lustre: server umount lustre-MDT0000 complete [ 6089.501793] LustreError: 423509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760143657 with bad export cookie 17728049288400509595 [ 6089.506732] LustreError: 423509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6089.507598] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6089.571791] LustreError: 425943:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6089.573517] LustreError: 425943:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6089.647880] Lustre: server umount lustre-MDT0001 complete [ 6094.471354] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6096.107434] Key type lgssc unregistered [ 6096.271773] LNet: 426474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6096.276809] LNetError: 426474:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6096.287722] LNet: Removed LNI 192.168.203.145@tcp [ 6096.782175] Key type .llcrypt unregistered [ 6096.784310] Key type ._llcrypt unregistered [ 6108.313535] Key type ._llcrypt registered [ 6108.315553] Key type .llcrypt registered [ 6108.386391] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6108.850210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6108.894491] alg: No test for adler32 (adler32-zlib) [ 6109.797875] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 6109.906144] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 6111.504255] Key type lgssc registered [ 6112.165317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6115.756631] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6115.768135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6131.232340] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6136.352267] LustreError: 427829:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [ 6162.976128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6164.655965] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6166.181171] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6168.165188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6168.322303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6168.343718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6169.482068] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6171.992940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6172.118158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6173.268490] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6174.817496] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6175.887256] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6182.373646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6182.375146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6182.378157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6182.381047] Lustre: Skipped 1 previous similar message [ 6182.383526] Lustre: Skipped 1 previous similar message [ 6187.493110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6187.495812] Lustre: Skipped 1 previous similar message [ 6188.397109] LustreError: 430110:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 6188.458405] Lustre: server umount lustre-OST0000 complete [ 6192.610971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6192.612039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6192.614027] Lustre: Skipped 1 previous similar message [ 6192.616684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6195.707202] LustreError: 430313:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6195.708847] LustreError: 430313:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6195.734396] Lustre: server umount lustre-MDT0000 complete [ 6196.849606] LustreError: 427838:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760143764 with bad export cookie 6269077670587927686 [ 6196.854584] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6196.918076] LustreError: 430515:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6196.919947] LustreError: 430515:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6197.011046] Lustre: server umount lustre-MDT0001 complete [ 6200.412213] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6201.414213] Key type lgssc unregistered [ 6201.566390] LNet: 431046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6201.568552] LNetError: 431046:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6201.577400] LNet: Removed LNI 192.168.203.145@tcp [ 6201.880698] Key type .llcrypt unregistered [ 6201.881682] Key type ._llcrypt unregistered [ 6210.520995] Key type ._llcrypt registered [ 6210.521910] Key type .llcrypt registered [ 6210.561094] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6210.968105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6211.029589] alg: No test for adler32 (adler32-zlib) [ 6211.876806] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 6211.950322] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 6213.528082] Key type lgssc registered [ 6213.869436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6216.092030] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6216.097578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6217.239370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6218.273925] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6220.882800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6221.004710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6222.218895] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6223.786644] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6224.841240] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6226.830294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6226.938250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6227.948094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6228.619921] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6230.194502] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6236.131093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6236.133736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6236.134155] Lustre: Skipped 2 previous similar messages [ 6236.135774] Lustre: Skipped 1 previous similar message [ 6238.017293] LustreError: 434665:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6238.044961] Lustre: server umount lustre-MDT0000 complete [ 6238.754514] LustreError: 432402:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.45@tcp arrived at 1760143806 with bad export cookie 8782080066889699117 [ 6238.758441] LustreError: 432402:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6239.343062] LustreError: 432402:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760143807 with bad export cookie 8782080066889698242 [ 6239.346686] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6241.248977] LustreError: 432421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6241.249592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6241.249712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6241.250212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6241.250216] Lustre: Skipped 1 previous similar message [ 6241.252650] LustreError: 432421:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6244.256730] LustreError: 432421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6244.256926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6244.260097] LustreError: 432421:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6245.418202] LustreError: 434868:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6245.419693] LustreError: 434868:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6245.468996] Lustre: server umount lustre-OST0000 complete [ 6247.293374] LustreError: 435221:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6247.296027] LustreError: 435221:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6247.370555] Lustre: server umount lustre-MDT0001 complete [ 6250.334297] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6251.183808] Key type lgssc unregistered [ 6251.294299] LNet: 435753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6251.296057] LNetError: 435753:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6251.304231] LNet: Removed LNI 192.168.203.145@tcp [ 6251.519665] Key type .llcrypt unregistered [ 6251.520501] Key type ._llcrypt unregistered [ 6259.514900] Key type ._llcrypt registered [ 6259.516105] Key type .llcrypt registered [ 6259.554957] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6260.059320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6260.071730] alg: No test for adler32 (adler32-zlib) [ 6260.929913] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 6261.011522] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 6262.584099] Key type lgssc registered [ 6262.967266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6265.637850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6265.644812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6266.807889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6268.030711] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6271.247155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6271.434332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6273.281599] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6275.920703] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6277.398955] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6280.700343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6280.903135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6283.365956] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6286.003779] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6292.205571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6301.623914] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 6302.177739] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6302.182289] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6302.190468] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6302.696975] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6303.948252] LustreError: 439392:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6303.986539] Lustre: server umount lustre-MDT0001 complete [ 6307.813558] LustreError: 438332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6307.814182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6307.822923] LustreError: 438332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6307.830531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6307.831156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6307.831167] Lustre: Skipped 1 previous similar message [ 6312.444895] LustreError: 439596:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6312.448785] LustreError: 439596:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6312.551594] Lustre: server umount lustre-OST0000 complete [ 6312.932894] LustreError: 438332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6314.488286] LustreError: 439797:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6314.491988] LustreError: 439797:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6314.629722] Lustre: server umount lustre-MDT0000 complete [ 6321.285694] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6322.947363] Key type lgssc unregistered [ 6323.140581] LNet: 440480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6323.145464] LNetError: 440480:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6323.156591] LNet: Removed LNI 192.168.203.145@tcp [ 6323.650227] Key type .llcrypt unregistered [ 6323.652040] Key type ._llcrypt unregistered [ 6335.378153] Key type ._llcrypt registered [ 6335.379484] Key type .llcrypt registered [ 6335.441664] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 6343.673748] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6344.143834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6344.185977] alg: No test for adler32 (adler32-zlib) [ 6345.088847] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 6345.197932] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 6346.800205] Key type lgssc registered [ 6347.367232] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6351.917234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6355.239793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6357.700864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6360.471088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6363.451685] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6366.676639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6368.612268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6368.633178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6368.637452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6369.712595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6369.722625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6369.757945] Lustre: lustre-MDT0000: new disk, initializing [ 6369.779845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6369.785500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6370.880663] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6374.062592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6375.945929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6375.969207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6376.050108] Lustre: lustre-OST0000: new disk, initializing [ 6376.051878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6376.053389] Lustre: Skipped 1 previous similar message [ 6376.072558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6376.482195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6376.484780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6376.492095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6377.634536] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6380.378509] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6383.311037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6385.229590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6385.251254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6385.280187] Lustre: 444428:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6385.303320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6385.337222] Lustre: lustre-MDT0001: new disk, initializing [ 6385.373745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6385.385378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6385.387877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6386.459965] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6389.575976] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6389.637188] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6390.735387] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6392.836981] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [ 6396.898463] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6396.898853] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6396.901472] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6396.904012] Lustre: Skipped 1 previous similar message [ 6398.383829] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 6398.385374] Lustre: Skipped 1 previous similar message [ 6400.247680] LustreError: 447859:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6400.265555] Lustre: server umount lustre-MDT0001 complete [ 6407.138188] LustreError: 444433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6407.142009] LustreError: 444433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6412.257656] LustreError: 444439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6412.257756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6412.257774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6412.258223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6417.377399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6417.377989] LustreError: 444439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6418.219097] LustreError: 448131:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6418.220766] LustreError: 448131:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6418.278753] Lustre: server umount lustre-OST0000 complete [ 6425.529767] LustreError: 448332:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6425.531514] LustreError: 448332:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6425.608460] Lustre: server umount lustre-MDT0000 complete [ 6429.512330] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6430.395949] Key type lgssc unregistered [ 6430.512339] LNet: 449016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6430.514310] LNetError: 449016:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6430.523303] LNet: Removed LNI 192.168.203.145@tcp [ 6430.789994] Key type .llcrypt unregistered [ 6430.790972] Key type ._llcrypt unregistered [ 6447.743886] Key type ._llcrypt registered [ 6447.745122] Key type .llcrypt registered [ 6447.791800] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6447.930249] Key type .llcrypt unregistered [ 6447.931423] Key type ._llcrypt unregistered [ 6453.764217] Key type ._llcrypt registered [ 6453.764944] Key type .llcrypt registered [ 6453.803263] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6454.140614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6454.196072] alg: No test for adler32 (adler32-zlib) [ 6455.042998] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 6455.119200] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 6456.696107] Key type lgssc registered [ 6457.002262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6459.166469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6460.955835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6462.669026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6464.603599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6464.623641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6464.627782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6465.702629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6465.712262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6465.745827] Lustre: lustre-MDT0000: new disk, initializing [ 6465.767257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6465.772641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6466.863246] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6470.780287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6470.808647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6470.837856] Lustre: 452332:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6470.858174] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6470.859701] Lustre: Skipped 1 previous similar message [ 6470.893436] Lustre: lustre-MDT0001: new disk, initializing [ 6470.923668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6470.933730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6470.936462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6472.051256] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6474.839888] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6475.868562] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6477.901570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6477.920113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6478.004379] Lustre: lustre-OST0000: new disk, initializing [ 6478.005996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6478.029500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6479.683798] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6481.842481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6481.844852] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6481.869188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6482.519602] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6486.496940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6486.499214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6486.503689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6487.009426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6487.010217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6487.013896] Lustre: Skipped 1 previous similar message [ 6490.827405] LustreError: 454774:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6490.866314] Lustre: server umount lustre-MDT0000 complete [ 6492.636066] LustreError: 452324:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760144060 with bad export cookie 16979075355314759202 [ 6492.641748] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6492.642801] LustreError: 452324:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6492.732155] LustreError: 454978:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6492.735533] LustreError: 454978:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6492.833247] Lustre: server umount lustre-MDT0001 complete [ 6504.631529] LustreError: 455178:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6504.635406] LustreError: 455178:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6504.740835] Lustre: server umount lustre-OST0000 complete [ 6509.033488] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:54:35 (1760144075) [ 6509.519476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6512.678500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6512.832725] LustreError: 456005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6512.838541] LustreError: 456005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6512.854109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6514.228065] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6518.243307] LustreError: 456005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6518.320213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6518.504258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6519.672767] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6521.936879] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6523.440828] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6526.184044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6526.216330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6526.296976] Lustre: Found index 0 for lustre-OST0000, updating log [ 6528.322913] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6530.075265] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6537.639061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6542.818255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6542.819470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6542.820647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6542.823246] Lustre: Skipped 1 previous similar message [ 6542.824671] Lustre: Skipped 2 previous similar messages [ 6543.786673] LustreError: 458282:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6543.788261] LustreError: 458282:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6543.843268] Lustre: server umount lustre-OST0000 complete [ 6544.638451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6547.279404] Lustre: server umount lustre-MDT0000 complete [ 6548.334118] LustreError: 457621:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760144116 with bad export cookie 16979075355314760581 [ 6548.337079] LustreError: 457621:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6548.337587] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6548.401373] LustreError: 458738:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6548.403086] LustreError: 458738:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6548.475578] Lustre: server umount lustre-MDT0001 complete [ 6550.816573] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:55:17 (1760144117) [ 6551.111752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6551.400395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6551.683482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6551.968475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6552.250345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6552.540092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6552.829321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6553.112320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6553.396170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6553.679150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6553.967243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6554.253403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6554.539299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6554.826144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6555.105461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6555.387249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6555.675527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6555.966420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6556.248161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6556.531311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6556.817133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6557.097671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6557.384229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6557.682364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6557.967166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6558.247310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6558.547192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6558.840367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6559.125033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6559.416206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6559.710320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6559.998289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6560.291274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6560.590173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6560.886367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6561.176359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6561.473938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6561.774347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6562.065179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6562.349205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6562.639560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6562.928571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6563.208405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6563.505747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6563.800298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6564.083034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6564.372150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6564.657144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6564.942160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6565.222279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6565.510200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6567.696187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6567.833766] LustreError: 462118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6567.854443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6567.856025] Lustre: Skipped 1 previous similar message [ 6568.948469] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6571.391898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6572.592076] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6574.252711] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6575.314865] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6577.252627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6577.278246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6577.354642] Lustre: Found index 0 for lustre-OST0000, updating log [ 6579.005262] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6579.494506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6580.548153] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6582.178629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6582.179434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6582.183332] Lustre: Skipped 1 previous similar message [ 6587.244098] LustreError: 464433:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6587.245875] LustreError: 464433:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6587.300239] Lustre: server umount lustre-OST0000 complete [ 6589.473537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6589.474718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6589.477045] Lustre: Skipped 1 previous similar message [ 6594.593371] LustreError: 462118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6594.598904] LustreError: 462118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6594.605986] Lustre: server umount lustre-MDT0000 complete [ 6596.436567] LustreError: 462098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760144164 with bad export cookie 16979075355314761652 [ 6596.442057] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6596.442174] LustreError: 462098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6596.607866] Lustre: server umount lustre-MDT0001 complete [ 6597.701770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6605.026136] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 6609.612677] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6613.809294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6617.124627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6620.105318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6623.193290] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6626.548625] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:56:33 (1760144193) [ 6663.486487] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6664.981901] Key type lgssc unregistered [ 6665.142699] LNet: 468424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6665.147487] LNetError: 468424:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6793.191819] LNet: Removed LNI 192.168.203.145@tcp [ 6793.658237] Key type .llcrypt unregistered [ 6793.660241] Key type ._llcrypt unregistered [ 6801.366773] Key type ._llcrypt registered [ 6801.368814] Key type .llcrypt registered [ 6801.430928] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6801.752219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6801.792228] alg: No test for adler32 (adler32-zlib) [ 6802.651027] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 6802.741409] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 6804.328139] Key type lgssc registered [ 6804.832252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6806.253973] LNet: Added LNI 192.168.203.145@tcp100 [8/256/0/180] [ 6807.008486] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [ 6814.663436] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6815.724685] Key type lgssc unregistered [ 6815.864774] LNet: 470290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6815.866715] LNetError: 470290:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6815.869487] LNet: Removed LNI 192.168.203.145@tcp [ 6815.874516] LNet: Removed LNI 192.168.203.145@tcp100 [ 6816.264156] Key type .llcrypt unregistered [ 6816.266241] Key type ._llcrypt unregistered [ 6825.456887] Key type ._llcrypt registered [ 6825.457874] Key type .llcrypt registered [ 6825.489183] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 6832.544432] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6832.912881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6832.920583] alg: No test for adler32 (adler32-zlib) [ 6833.811818] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 6833.909437] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 6835.496099] Key type lgssc registered [ 6835.801427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6839.654533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6842.697325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6845.665550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6847.810297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6850.947792] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 21:00:17 (1760144417) [ 6858.106388] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6859.263377] Key type lgssc unregistered [ 6859.424626] LNet: 474302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6859.429339] LNetError: 474302:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6859.439517] LNet: Removed LNI 192.168.203.145@tcp [ 6859.887673] Key type .llcrypt unregistered [ 6859.889708] Key type ._llcrypt unregistered [ 6867.672469] Key type ._llcrypt registered [ 6867.674228] Key type .llcrypt registered [ 6867.722472] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6868.045199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6868.078051] alg: No test for adler32 (adler32-zlib) [ 6868.939578] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 6869.026525] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 6870.600206] Key type lgssc registered [ 6871.186276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6875.666967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6875.700518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6875.710343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6876.800564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6876.810317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6876.844097] Lustre: lustre-MDT0000: new disk, initializing [ 6876.865748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6876.871148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6878.348490] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6883.736633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6883.773598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6883.824770] Lustre: 475473:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6883.858574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6883.861154] Lustre: Skipped 1 previous similar message [ 6883.901376] Lustre: lustre-MDT0001: new disk, initializing [ 6883.942826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6883.961909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6883.966806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6885.707347] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6889.295792] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6890.971351] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6894.176563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6894.225375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6894.383807] Lustre: lustre-OST0000: new disk, initializing [ 6894.387536] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6894.442676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6895.051765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6895.056654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6895.113661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6897.121760] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6901.048285] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6902.731515] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing lnet_if_list [ 6905.061204] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 6905.080886] LNet: Added LNI 192.168.203.145@tcp42 [8/256/0/180] [ 6915.558980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6915.559194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6915.563281] LustreError: Skipped 1 previous similar message [ 6915.564265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6915.570140] Lustre: Skipped 1 previous similar message [ 6919.152235] LustreError: 478864:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6919.252265] Lustre: server umount lustre-OST0000 complete [ 6924.769642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6924.774083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6924.781869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6924.785473] Lustre: Skipped 1 previous similar message [ 6925.794390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6925.796126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6927.180321] LustreError: 479069:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6927.184213] LustreError: 479069:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6927.228536] Lustre: server umount lustre-MDT0000 complete [ 6929.031929] LustreError: 475465:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760144496 with bad export cookie 9728278700547570184 [ 6929.037378] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6929.038520] LustreError: 475465:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6929.116185] LustreError: 479271:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6929.117773] LustreError: 479271:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6929.229035] Lustre: server umount lustre-MDT0001 complete [ 6933.899533] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6935.333288] Key type lgssc unregistered [ 6935.547440] LNet: 479809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6935.552187] LNetError: 479809:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6935.556746] LNet: Removed LNI 192.168.203.145@tcp [ 6935.566571] LNet: Removed LNI 192.168.203.145@tcp42 [ 6935.975442] Key type .llcrypt unregistered [ 6935.977118] Key type ._llcrypt unregistered [ 6945.196015] Key type ._llcrypt registered [ 6945.198050] Key type .llcrypt registered [ 6945.267500] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6945.737900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6945.770985] alg: No test for adler32 (adler32-zlib) [ 6946.657154] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 6946.777327] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 6948.392165] Key type lgssc registered [ 6948.921136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6952.218376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6952.230774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6953.460931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6955.378985] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6959.530171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6959.732832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6961.573545] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6964.110927] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6965.813175] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6969.113832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6969.305433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6972.058518] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6974.615690] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6984.595805] loop0: detected capacity change from 0 to 204800000 [ 6984.688307] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6984.691886] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6984.696294] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6984.700743] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6984.727635] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6984.754283] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6984.773551] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6984.801336] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6984.834835] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6984.874683] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6987.793242] loop0: detected capacity change from 0 to 204800000 [ 6987.899159] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6987.962085] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6990.360601] loop0: detected capacity change from 0 to 204800000 [ 6992.909975] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6992.943802] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6993.004369] Lustre: fs15246-OST0000: new disk, initializing [ 6993.007741] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 6993.037392] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6995.530674] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7000.609345] loop1: detected capacity change from 0 to 204800000 [ 7003.286954] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7003.313864] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7003.364148] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7003.399521] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7003.452212] Lustre: fs15246-MDT0000: new disk, initializing [ 7003.528281] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7003.543652] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7003.602952] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7004.640701] Lustre: 485407:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@00000000ceefc02c x1845645293945088/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1760144588 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7005.267580] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7013.732444] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7013.738180] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7013.752364] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7020.010185] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7020.018545] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7021.447147] LustreError: 486534:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 7021.485364] Lustre: server umount fs15246-MDT0000 complete [ 7023.016308] LustreError: 481054:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7023.026187] LustreError: 481054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7025.688117] LustreError: 487036:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 7025.692053] LustreError: 487036:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7025.714093] Lustre: server umount fs15246-OST0000 complete [ 7031.777257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7031.781965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7031.789940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7033.318097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7033.318648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7036.594132] LustreError: 487537:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7036.597913] LustreError: 487537:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7036.690509] Lustre: server umount lustre-OST0000 complete [ 7041.504807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7041.508979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7041.515946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7041.517893] Lustre: Skipped 1 previous similar message [ 7044.623940] LustreError: 487739:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7044.628134] LustreError: 487739:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7044.661322] Lustre: server umount lustre-MDT0000 complete [ 7046.412099] LustreError: 482650:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760144614 with bad export cookie 11075696755702014599 [ 7046.417409] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7046.421445] LustreError: 482650:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7046.593948] Lustre: server umount lustre-MDT0001 complete [ 7051.070941] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7052.541487] Key type lgssc unregistered [ 7052.710728] LNet: 488471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7052.715366] LNetError: 488471:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7052.729519] LNet: Removed LNI 192.168.203.145@tcp [ 7053.205888] Key type .llcrypt unregistered [ 7053.208043] Key type ._llcrypt unregistered [ 7062.364576] Key type ._llcrypt registered [ 7062.366283] Key type .llcrypt registered [ 7062.423503] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7062.819253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7062.847333] alg: No test for adler32 (adler32-zlib) [ 7063.706958] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 7063.807651] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 7065.384166] Key type lgssc registered [ 7065.979412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7069.613349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7072.016105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7074.559242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7076.842247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7084.121126] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 7088.717929] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7093.514994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7096.946928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7099.948346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7103.104055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7106.375913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7106.405843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7106.416456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7107.518891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7107.533376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7107.581216] Lustre: lustre-MDT0000: new disk, initializing [ 7107.609240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7107.616664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7109.296483] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7114.744820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7114.782545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7114.840725] Lustre: 492876:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7114.882475] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7114.886066] Lustre: Skipped 1 previous similar message [ 7114.937704] Lustre: lustre-MDT0001: new disk, initializing [ 7114.992733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7115.014387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7115.020677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7116.937606] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7121.107357] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7122.990595] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7126.381352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7126.432181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7126.602927] Lustre: lustre-OST0000: new disk, initializing [ 7126.606713] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7126.662333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7129.483421] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7133.642670] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7134.228243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7134.233945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7134.297698] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7135.690063] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7135.824187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7137.807981] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7137.948981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7139.302628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7139.302953] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7139.307763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7139.309156] LustreError: Skipped 1 previous similar message [ 7139.311971] Lustre: Skipped 1 previous similar message [ 7140.915159] LustreError: 495816:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7141.020777] Lustre: server umount lustre-OST0000 complete [ 7144.420817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7144.421887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7144.428186] Lustre: Skipped 2 previous similar messages [ 7144.433894] Lustre: Skipped 1 previous similar message [ 7149.538462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7149.542203] Lustre: Skipped 1 previous similar message [ 7154.658722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7154.662377] Lustre: Skipped 1 previous similar message [ 7157.216152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7157.269349] LustreError: 496016:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7157.273253] LustreError: 496016:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7157.324877] Lustre: server umount lustre-MDT0000 complete [ 7159.172957] LustreError: 492868:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760144726 with bad export cookie 10375366931921262202 [ 7159.179987] LustreError: 492868:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7159.180452] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7159.276259] LustreError: 496219:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7159.279766] LustreError: 496219:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7159.375416] Lustre: server umount lustre-MDT0001 complete [ 7171.036960] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 7175.878284] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7181.131410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7185.026338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7188.019664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7191.338884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7192.210185] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7057 sec ========== 21:05:58 (1760144758) [ 7193.075480] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:05:59 (1760144759) === [ 7194.757718] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:06:01 (1760144761) === [ 7214.015778] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7215.692104] Key type lgssc unregistered [ 7215.846705] LNet: 501071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7215.851376] LNetError: 501071:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7215.862793] LNet: Removed LNI 192.168.203.145@tcp [ 7216.374103] Key type .llcrypt unregistered [ 7216.376205] Key type ._llcrypt unregistered