[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 450379457 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003057] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005008] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007015] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008006] pid_max: default: 32768 minimum: 301 [ 0.009114] LSM: Security Framework initializing [ 0.010032] Yama: becoming mindful. [ 0.011022] SELinux: Initializing. [ 0.012056] *** VALIDATE selinux *** [ 0.019704] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023360] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024129] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025082] *** VALIDATE tmpfs *** [ 0.027151] *** VALIDATE proc *** [ 0.028166] *** VALIDATE cgroup *** [ 0.029004] *** VALIDATE cgroup2 *** [ 0.030142] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031131] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033022] Spectre V2 : User space: Vulnerable [ 0.034004] Speculative Store Bypass: Vulnerable [ 0.037558] debug: unmapping init [mem 0xffffffffb1e59000-0xffffffffb1e60fff] [ 0.040928] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041833] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042017] ... version: 2 [ 0.043018] ... bit width: 48 [ 0.044015] ... generic registers: 4 [ 0.045012] ... value mask: 0000ffffffffffff [ 0.046014] ... max period: 00007fffffffffff [ 0.047011] ... fixed-purpose events: 3 [ 0.048009] ... event mask: 000000070000000f [ 0.049332] rcu: Hierarchical SRCU implementation. [ 0.051520] smp: Bringing up secondary CPUs ... [ 0.052551] x86: Booting SMP configuration: [ 0.053027] .... node #0, CPUs: #1 #2 #3 [ 0.060787] smp: Brought up 1 node, 4 CPUs [ 0.063018] smpboot: Max logical packages: 1 [ 0.064014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.092556] node 0 deferred pages initialised in 26ms [ 0.095414] devtmpfs: initialized [ 0.096273] x86/mm: Memory block size: 128MB [ 0.099988] gcov: version magic: 0x41383552 [ 0.102306] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.103000] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.104418] pinctrl core: initialized pinctrl subsystem [ 0.106202] [ 0.106720] ************************************************************* [ 0.109019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.111014] ** ** [ 0.113017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.115014] ** ** [ 0.117018] ** This means that this kernel is built to expose internal ** [ 0.120017] ** IOMMU data structures, which may compromise security on ** [ 0.122012] ** your system. ** [ 0.124014] ** ** [ 0.126018] ** If you see this message and you are not debugging the ** [ 0.129016] ** kernel, report this immediately to your vendor! ** [ 0.131012] ** ** [ 0.133019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.135016] ************************************************************* [ 0.137975] NET: Registered protocol family 16 [ 0.140494] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.143079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.145131] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.149115] cpuidle: using governor menu [ 0.153251] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.155682] PCI: Using configuration type 1 for base access [ 0.157155] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.166259] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.167248] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.172085] cryptd: max_cpu_qlen set to 1000 [ 0.177456] ACPI: Added _OSI(Module Device) [ 0.179023] ACPI: Added _OSI(Processor Device) [ 0.180035] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.182016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.186000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.200000] ACPI: Interpreter enabled [ 0.201066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203017] ACPI: Using IOAPIC for interrupt routing [ 0.204124] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.207585] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.216956] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.218050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.220025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.223107] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.227345] acpiphp: Slot [2] registered [ 0.229110] acpiphp: Slot [3] registered [ 0.230136] acpiphp: Slot [4] registered [ 0.232165] acpiphp: Slot [5] registered [ 0.233134] acpiphp: Slot [6] registered [ 0.234108] acpiphp: Slot [7] registered [ 0.235137] acpiphp: Slot [8] registered [ 0.236000] acpiphp: Slot [9] registered [ 0.236000] acpiphp: Slot [10] registered [ 0.238111] acpiphp: Slot [11] registered [ 0.240112] acpiphp: Slot [12] registered [ 0.241085] acpiphp: Slot [13] registered [ 0.242118] acpiphp: Slot [14] registered [ 0.244136] acpiphp: Slot [15] registered [ 0.245000] acpiphp: Slot [16] registered [ 0.245000] acpiphp: Slot [17] registered [ 0.245214] acpiphp: Slot [18] registered [ 0.246000] acpiphp: Slot [19] registered [ 0.246120] acpiphp: Slot [20] registered [ 0.247000] acpiphp: Slot [21] registered [ 0.247087] acpiphp: Slot [22] registered [ 0.248000] acpiphp: Slot [23] registered [ 0.249107] acpiphp: Slot [24] registered [ 0.250093] acpiphp: Slot [25] registered [ 0.251000] acpiphp: Slot [26] registered [ 0.251000] acpiphp: Slot [27] registered [ 0.253143] acpiphp: Slot [28] registered [ 0.254086] acpiphp: Slot [29] registered [ 0.255091] acpiphp: Slot [30] registered [ 0.258244] acpiphp: Slot [31] registered [ 0.259070] PCI host bridge to bus 0000:00 [ 0.260019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.264040] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.269026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.271037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.275037] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.278033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.282401] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.288611] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.294000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.303000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.319071] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.321026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.332022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.336023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.339327] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.341857] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.348054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.351262] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.356019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.365021] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.368932] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.375000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.385024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.395029] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.408023] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.419887] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.428019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.437034] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.462024] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.478689] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.488024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.504019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.524023] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.546874] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.558026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.562026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.583022] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.592000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.606021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.612019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.629021] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.645170] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.652019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.661021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.682019] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.697277] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.699358] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.701443] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.702000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.702210] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.710043] iommu: Default domain type: Passthrough [ 0.712493] SCSI subsystem initialized [ 0.714112] ACPI: bus type USB registered [ 0.716213] usbcore: registered new interface driver usbfs [ 0.718086] usbcore: registered new interface driver hub [ 0.720085] usbcore: registered new device driver usb [ 0.722132] pps_core: LinuxPPS API ver. 1 registered [ 0.723007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.724104] PTP clock support registered [ 0.727044] EDAC MC: Ver: 3.0.0 [ 0.729618] PCI: Using ACPI for IRQ routing [ 0.733399] NetLabel: Initializing [ 0.734014] NetLabel: domain hash size = 128 [ 0.736009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.738191] NetLabel: unlabeled traffic allowed by default [ 0.743069] vgaarb: loaded [ 0.744000] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.744019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.760683] clocksource: Switched to clocksource kvm-clock [ 0.877507] VFS: Disk quotas dquot_6.6.0 [ 0.879313] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.883327] *** VALIDATE ramfs *** [ 0.884486] *** VALIDATE hugetlbfs *** [ 0.885838] pnp: PnP ACPI init [ 0.888154] pnp: PnP ACPI: found 6 devices [ 0.907215] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.909958] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.912077] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.913665] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.915332] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.917747] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.919898] NET: Registered protocol family 2 [ 0.921750] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.926554] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.930074] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.935075] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.938298] TCP: Hash tables configured (established 65536 bind 65536) [ 0.943304] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.946364] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.949040] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.953482] NET: Registered protocol family 1 [ 0.959294] RPC: Registered named UNIX socket transport module. [ 0.960868] RPC: Registered udp transport module. [ 0.962166] RPC: Registered tcp transport module. [ 0.964299] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.966842] NET: Registered protocol family 44 [ 0.968047] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.969532] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.971740] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.973873] PCI: CLS 0 bytes, default 64 [ 0.975687] Unpacking initramfs... [ 3.314725] debug: unmapping init [mem 0xffff9d5ffcc54000-0xffff9d5ffffbffff] [ 3.321714] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.324043] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.327790] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.931337] Initialise system trusted keyrings [ 3.932910] Key type blacklist registered [ 3.935411] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.944643] zbud: loaded [ 3.947924] *** VALIDATE nfs *** [ 3.948987] *** VALIDATE nfs4 *** [ 3.950641] pstore: using deflate compression [ 3.954336] Platform Keyring initialized [ 4.157572] NET: Registered protocol family 38 [ 4.162195] Key type asymmetric registered [ 4.163699] Asymmetric key parser 'x509' registered [ 4.181695] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.186455] io scheduler mq-deadline registered [ 4.188915] io scheduler kyber registered [ 4.190313] io scheduler bfq registered [ 4.197246] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.200143] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.212478] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.216719] ACPI: Power Button [PWRF] [ 4.345911] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.454259] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.684463] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.860551] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.185910] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.223957] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.265085] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.271961] Non-volatile memory driver v1.3 [ 5.273865] Linux agpgart interface v0.103 [ 5.315789] virtio_blk virtio1: [vda] 133600 512-byte logical blocks (68.4 MB/65.2 MiB) [ 5.323435] vda: detected capacity change from 0 to 68403200 [ 5.346464] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.349083] vdb: detected capacity change from 0 to 1073741824 [ 5.369419] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.372135] vdc: detected capacity change from 0 to 2621440000 [ 5.392968] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.397191] vdd: detected capacity change from 0 to 2621440000 [ 5.420357] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.427205] vde: detected capacity change from 0 to 4294967296 [ 5.455970] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.458924] vdf: detected capacity change from 0 to 4294967296 [ 5.468107] libphy: Fixed MDIO Bus: probed [ 5.475774] usbcore: registered new interface driver usbserial_generic [ 5.478761] usbserial: USB Serial support registered for generic [ 5.481829] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.487148] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.488972] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.492210] mousedev: PS/2 mouse device common for all mice [ 5.495448] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.496313] rtc_cmos 00:05: RTC can wake from S4 [ 5.509516] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.514193] rtc_cmos 00:05: registered as rtc0 [ 5.514243] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.514277] intel_pstate: CPU model not supported [ 5.522460] hid: raw HID events driver (C) Jiri Kosina [ 5.522701] usbcore: registered new interface driver usbhid [ 5.522705] usbhid: USB HID core driver [ 5.522811] drop_monitor: Initializing network drop monitor service [ 5.522936] Initializing XFRM netlink socket [ 5.523314] NET: Registered protocol family 10 [ 5.527430] Segment Routing with IPv6 [ 5.554598] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.571656] NET: Registered protocol family 17 [ 5.574520] mpls_gso: MPLS GSO support [ 5.583712] RAS: Correctable Errors collector initialized. [ 5.585734] AVX version of gcm_enc/dec engaged. [ 5.587283] AES CTR mode by8 optimization enabled [ 5.752776] sched_clock: Marking stable (5752410408, 0)->(6844005958, -1091595550) [ 5.762552] registered taskstats version 1 [ 5.770123] Loading compiled-in X.509 certificates [ 5.774239] zswap: loaded using pool lzo/zbud [ 5.815627] Key type big_key registered [ 5.830873] Key type encrypted registered [ 5.832871] ima: No TPM chip found, activating TPM-bypass! [ 5.838868] ima: Allocated hash algorithm: sha1 [ 5.841407] ima: No architecture policies found [ 5.843551] evm: Initialising EVM extended attributes: [ 5.845869] evm: security.selinux [ 5.847517] evm: security.ima [ 5.848450] evm: security.capability [ 5.849756] evm: HMAC attrs: 0x1 [ 5.854839] rtc_cmos 00:05: setting system clock to 2025-11-03 10:39:45 UTC (1762166385) [ 5.866883] debug: unmapping init [mem 0xffffffffb2e03000-0xffffffffb2ffffff] [ 5.891369] debug: unmapping init [mem 0xffffffffb1b82000-0xffffffffb1e58fff] [ 5.907225] Write protecting the kernel read-only data: 28672k [ 5.914159] debug: unmapping init [mem 0xffffffffb0203000-0xffffffffb03fffff] [ 5.920886] debug: unmapping init [mem 0xffffffffb0b14000-0xffffffffb0bfffff] [ 5.994188] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.012304] systemd[1]: Detected virtualization kvm. [ 6.016298] systemd[1]: Detected architecture x86-64. [ 6.020271] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.056579] systemd[1]: No hostname configured. [ 6.058840] systemd[1]: Set hostname to . [ 6.060619] random: systemd: uninitialized urandom read (16 bytes read) [ 6.062933] systemd[1]: Initializing machine ID from random generator. [ 6.170212] random: ln: uninitialized urandom read (6 bytes read) [ 6.506948] random: systemd: uninitialized urandom read (16 bytes read) [ 6.509928] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 6.519737] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 6.527599] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.680664] device-mapper: uevent: version 1.0.3 [ 8.682351] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.006212] virtio_net virtio0 ens2: renamed from eth0 [ 11.120066] random: fast init done [ 12.364102] scsi host0: ata_piix [ 12.570840] scsi host1: ata_piix [ 12.572150] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.574353] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.365356] random: crng init done [ 17.372419] random: 7 urandom warning(s) missed due to ratelimiting [ 20.346295] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.848168] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.335102] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.333085] SELinux: Disabled at runtime. [ 25.444515] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.466235] systemd[1]: Detected virtualization kvm. [ 25.476664] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.244785] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.265502] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.285598] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.292228] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.307414] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.329638] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.346491] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ 27.886739] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 28.354927] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.435424] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.436402] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.022837] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.237851] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 36.416121] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 37.505688] NFS: Registering the id_resolver key type [ 37.512030] Key type id_resolver registered [ 37.513539] Key type id_legacy registered [ **] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg237-server login: [ 73.713593] libcfs: loading out-of-tree module taints kernel. [ 73.729293] Key type ._llcrypt registered [ 73.730875] Key type .llcrypt registered [ 73.782415] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 82.895674] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 83.542306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 83.550712] alg: No test for adler32 (adler32-zlib) [ 84.613936] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 85.074060] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 86.727147] Key type lgssc registered [ 87.453918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.963155] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 97.535585] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.783335] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 105.621680] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 109.475631] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 117.789852] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 124.860871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 124.890254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 124.902118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 126.043211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 126.065099] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 126.109757] Lustre: lustre-MDT0000: new disk, initializing [ 126.186181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 126.196423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 128.204810] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 135.171936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 135.214451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 135.252681] Lustre: 6484:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 135.276511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 135.278711] Lustre: Skipped 1 previous similar message [ 135.328614] Lustre: lustre-MDT0001: new disk, initializing [ 135.374280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 135.393954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 135.399805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 137.326125] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 140.067108] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 144.958168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 144.999852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 145.144902] Lustre: lustre-OST0000: new disk, initializing [ 145.147669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 145.174196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 148.190094] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 151.023347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 151.026653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 151.043700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 154.641751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 154.670233] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 154.726647] Lustre: lustre-OST0001: new disk, initializing [ 154.729560] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 154.760544] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 157.637896] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.221701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 158.232798] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 158.264239] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 165.298189] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 171.194450] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 178.929525] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing check_logdir /tmp/testlogs/ [ 181.414100] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing yml_node [ 184.078080] Lustre: DEBUG MARKER: Client: 2.16.59.50 [ 185.460283] Lustre: DEBUG MARKER: MDS: 2.16.59.50 [ 186.944981] Lustre: DEBUG MARKER: OSS: 2.16.59.50 [ 187.841761] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Nov 3 05:42:46 EST 2025 [ 197.202702] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 198.142414] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 212.447684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 212.451195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 212.457047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 214.496734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 214.498016] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 214.502129] Lustre: Skipped 1 previous similar message [ 214.510991] Lustre: Skipped 1 previous similar message [ 219.616367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 219.623127] Lustre: Skipped 5 previous similar messages [ 224.737173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 224.741686] Lustre: Skipped 3 previous similar messages [ 225.247116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 225.299652] LustreError: 12181:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 225.338480] Lustre: server umount lustre-MDT0000 complete [ 227.349187] LustreError: 6475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762166606 with bad export cookie 6977759202916660866 [ 227.351928] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 227.353561] LustreError: 6475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 229.857031] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 229.862623] Lustre: Skipped 1 previous similar message [ 229.865695] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 229.868394] Lustre: Skipped 1 previous similar message [ 233.491301] LustreError: 12384:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 233.499225] LustreError: 12384:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 233.595225] Lustre: server umount lustre-MDT0001 complete [ 235.683102] LustreError: 12585:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 235.689643] LustreError: 12585:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 235.713489] Lustre: server umount lustre-OST0000 complete [ 237.993276] LustreError: 12786:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 237.995390] LustreError: 12786:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 238.044549] Lustre: server umount lustre-OST0001 complete [ 241.066699] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 245.496743] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 249.958867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 253.450235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 256.664401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 260.514272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 264.621921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 264.662292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 264.780257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 264.800944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 264.845417] Lustre: lustre-MDT0000: new disk, initializing [ 264.886773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 264.895418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 266.926442] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 272.761441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 272.791710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 272.837064] Lustre: 14967:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 272.877952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 272.881309] Lustre: Skipped 1 previous similar message [ 272.924478] Lustre: lustre-MDT0001: new disk, initializing [ 272.970798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 272.994636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 273.002343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 275.011903] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 279.376299] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 281.075380] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 284.088144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 284.116428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 286.042203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 286.098265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 287.251958] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 291.361102] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 293.245910] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 293.359675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 295.502383] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 295.616955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 301.538499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.541103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 301.544057] Lustre: Skipped 2 previous similar messages [ 301.544652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 301.544662] Lustre: Skipped 2 previous similar messages [ 301.548914] LustreError: Skipped 1 previous similar message [ 302.710159] LustreError: 17905:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 302.729019] LustreError: 17905:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 302.784681] Lustre: server umount lustre-OST0000 complete [ 306.658552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.663215] Lustre: Skipped 1 previous similar message [ 310.720695] Lustre: server umount lustre-MDT0000 complete [ 311.779922] LustreError: 15957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.787395] LustreError: 15957:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 312.685649] LustreError: 16715:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762166692 with bad export cookie 6977759202916662399 [ 312.690560] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 312.690869] LustreError: 16715:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 312.787511] LustreError: 18307:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 312.789612] LustreError: 18307:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 312.890768] Lustre: server umount lustre-MDT0001 complete [ 314.907972] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 05:44:53 (1762166693) [ 316.193505] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 317.034934] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 05:44:55 (1762166695) [ 320.490446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 320.799098] LustreError: 18976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.841189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 320.844374] Lustre: Skipped 1 previous similar message [ 322.946054] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 326.112314] LustreError: 18976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 327.251682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 329.512764] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 332.278798] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 334.005165] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 337.280525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 339.924528] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 342.035824] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 353.760497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 353.763510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 353.772646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 353.774812] Lustre: Skipped 3 previous similar messages [ 355.810275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 356.850272] LustreError: 21226:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 356.853033] LustreError: 21226:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 356.911440] Lustre: server umount lustre-OST0000 complete [ 373.215151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 373.318059] Lustre: server umount lustre-MDT0000 complete [ 375.232400] LustreError: 18957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762166754 with bad export cookie 6977759202916663281 [ 375.236106] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 375.237107] LustreError: 18957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 375.406773] Lustre: server umount lustre-MDT0001 complete [ 380.400304] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 381.604693] Key type lgssc unregistered [ 381.758866] LNet: 22161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.762166] LNetError: 22161:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 381.773598] LNet: Removed LNI 192.168.202.137@tcp [ 382.201127] Key type .llcrypt unregistered [ 382.203829] Key type ._llcrypt unregistered [ 394.167979] Key type ._llcrypt registered [ 394.169716] Key type .llcrypt registered [ 394.234864] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 394.630924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 394.667656] alg: No test for adler32 (adler32-zlib) [ 395.573539] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 395.728223] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 397.337867] Key type lgssc registered [ 397.935625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 402.069044] loop: module loaded [ 402.092162] loop0: detected capacity change from 0 to 204800000 [ 402.186479] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 402.193023] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 402.362295] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 402.389142] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 405.627627] loop0: detected capacity change from 0 to 204800000 [ 405.717904] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 405.727201] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 405.736872] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 405.812151] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 408.345609] loop0: detected capacity change from 0 to 204800000 [ 411.355819] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 411.383951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 411.391873] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.521277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 412.550114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 412.592389] Lustre: lustre-MDT0000: new disk, initializing [ 412.633768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 412.645533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 412.662759] Lustre: 24272:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 466, rollback = 2 [ 412.667208] Lustre: 24272:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 412.671716] Lustre: 24272:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 2/466/0 [ 412.675496] Lustre: 24272:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 412.680183] Lustre: 24272:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 412.684426] Lustre: 24272:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 414.818541] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 420.072393] loop1: detected capacity change from 0 to 204800000 [ 422.688431] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 422.717167] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 422.854710] Lustre: lustre-OST07c6: new disk, initializing [ 422.860000] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 422.863908] Lustre: Skipped 1 previous similar message [ 422.900111] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 425.644385] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 431.119196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 431.126165] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 431.143138] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 434.701229] Lustre: 24297:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 434.705452] Lustre: 24297:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 1 previous similar message [ 434.708455] Lustre: 24297:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 434.710932] Lustre: 24297:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 434.714044] Lustre: 24297:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 434.717268] Lustre: 24297:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 434.721407] Lustre: 24297:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 434.726284] Lustre: 24297:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 434.731040] Lustre: 24297:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 434.734968] Lustre: 24297:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 434.738367] Lustre: 24297:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 434.743528] Lustre: 24297:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 449.493629] Lustre: 24297:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 497, rollback = 2 [ 449.497549] Lustre: 24297:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 449.501286] Lustre: 24297:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 449.504840] Lustre: 24297:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 449.508876] Lustre: 24297:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 449.513459] Lustre: 24297:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 453.314675] Lustre: 24297:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 497, rollback = 2 [ 453.320551] Lustre: 24297:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 1 previous similar message [ 453.323810] Lustre: 24297:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 453.327412] Lustre: 24297:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 453.331147] Lustre: 24297:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 453.334419] Lustre: 24297:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 453.338098] Lustre: 24297:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 453.342513] Lustre: 24297:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 453.345514] Lustre: 24297:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 453.349134] Lustre: 24297:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 453.351938] Lustre: 24297:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 453.355614] Lustre: 24297:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 460.081406] Lustre: 24297:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 497, rollback = 2 [ 460.084899] Lustre: 24297:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 3 previous similar messages [ 460.089105] Lustre: 24297:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 460.092941] Lustre: 24297:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 460.095966] Lustre: 24297:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 460.100192] Lustre: 24297:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 460.104141] Lustre: 24297:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 460.108716] Lustre: 24297:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 460.111596] Lustre: 24297:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 460.113951] Lustre: 24297:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 460.116113] Lustre: 24297:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 460.119092] Lustre: 24297:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 469.795433] Lustre: 24296:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 497, rollback = 2 [ 469.799211] Lustre: 24296:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 5 previous similar messages [ 469.802540] Lustre: 24296:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 469.805646] Lustre: 24296:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 469.808960] Lustre: 24296:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 469.811851] Lustre: 24296:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 469.815547] Lustre: 24296:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 469.820677] Lustre: 24296:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 469.825278] Lustre: 24296:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 469.829937] Lustre: 24296:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 469.834557] Lustre: 24296:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 469.838139] Lustre: 24296:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 518.115165] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 518.115986] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.128597] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 520.432602] LustreError: 26487:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 520.481248] Lustre: server umount lustre-OST07c6 complete [ 524.938125] LustreError: 26989:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 524.940474] LustreError: 26989:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 525.013477] Lustre: server umount lustre-MDT0000 complete [ 531.996526] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 533.120329] Key type lgssc unregistered [ 533.254541] LNet: 27820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 533.258187] LNetError: 27820:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 533.268318] LNet: Removed LNI 192.168.202.137@tcp [ 533.612435] Key type .llcrypt unregistered [ 533.613726] Key type ._llcrypt unregistered [ 542.649286] Key type ._llcrypt registered [ 542.650828] Key type .llcrypt registered [ 542.741486] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 543.142809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 543.184534] alg: No test for adler32 (adler32-zlib) [ 544.054605] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 544.170731] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 545.775180] Key type lgssc registered [ 546.221347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 549.633667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 549.648355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.891670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 552.488890] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 556.681227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 556.862851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 558.677535] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 560.943737] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 562.279216] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 565.034639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 565.186103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.579697] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 569.672247] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 575.469104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 582.121909] LustreError: 31266:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 585.696418] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 585.702223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 585.703655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 585.708223] Lustre: Skipped 1 previous similar message [ 590.817509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 590.820356] Lustre: Skipped 2 previous similar messages [ 591.730827] LustreError: 31516:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 591.787013] Lustre: server umount lustre-OST0000 complete [ 595.936681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.937833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 595.940623] Lustre: Skipped 1 previous similar message [ 599.617199] LustreError: 31718:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 599.619105] LustreError: 31718:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 599.647321] Lustre: server umount lustre-MDT0000 complete [ 601.055910] LustreError: 29052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.062498] LustreError: 29052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 601.086206] LustreError: 29033:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762166980 with bad export cookie 6923998619574806454 [ 601.089505] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 601.090783] LustreError: 29033:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 601.176529] LustreError: 31920:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 601.178595] LustreError: 31920:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 601.241535] Lustre: server umount lustre-MDT0001 complete [ 605.147930] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 606.428837] Key type lgssc unregistered [ 606.568494] LNet: 32451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 606.571826] LNetError: 32451:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 606.578540] LNet: Removed LNI 192.168.202.137@tcp [ 607.084682] Key type .llcrypt unregistered [ 607.086580] Key type ._llcrypt unregistered [ 615.983138] Key type ._llcrypt registered [ 615.984919] Key type .llcrypt registered [ 616.039633] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 616.445946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 616.521669] alg: No test for adler32 (adler32-zlib) [ 617.412619] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 617.539027] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 619.135207] Key type lgssc registered [ 619.578253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 622.958894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 622.967467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 624.204718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 626.015126] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 629.818615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 629.973245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 631.821266] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 633.924388] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 635.340674] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 638.091992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 638.230454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 640.600283] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 643.026926] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 648.492489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 653.795842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 653.796804] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 653.797982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 653.801027] Lustre: Skipped 1 previous similar message [ 654.392103] LustreError: 35933:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 654.448139] Lustre: server umount lustre-OST0000 complete [ 657.984160] LustreError: 36134:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 657.987014] LustreError: 36134:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 658.017861] Lustre: server umount lustre-MDT0000 complete [ 658.912602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 658.913190] LustreError: 33676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 658.916694] Lustre: Skipped 1 previous similar message [ 658.923203] LustreError: 33676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 659.404425] LustreError: 33659:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762167039 with bad export cookie 17639105279456700466 [ 659.404708] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.408380] LustreError: 33659:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 659.480819] LustreError: 36336:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 659.483399] LustreError: 36336:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 659.542049] Lustre: server umount lustre-MDT0001 complete [ 663.482606] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 664.586550] Key type lgssc unregistered [ 664.721618] LNet: 36866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.725198] LNetError: 36866:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.733945] LNet: Removed LNI 192.168.202.137@tcp [ 665.051906] Key type .llcrypt unregistered [ 665.052782] Key type ._llcrypt unregistered [ 674.789457] Key type ._llcrypt registered [ 674.791012] Key type .llcrypt registered [ 674.829311] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 675.195671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 675.255925] alg: No test for adler32 (adler32-zlib) [ 676.126878] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 676.226993] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 677.823228] Key type lgssc registered [ 678.249257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 681.399476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 681.407288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 682.634787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 684.210650] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 687.509548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 687.662543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 689.323888] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 691.633975] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 693.366171] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 696.422302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 696.572973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 698.820926] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 700.969527] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 708.835991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 714.208715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 714.210720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 714.213726] Lustre: Skipped 1 previous similar message [ 716.684626] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.37@tcp (stopping) [ 716.687483] Lustre: Skipped 2 previous similar messages [ 719.329070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 719.331638] Lustre: Skipped 2 previous similar messages [ 720.256396] LustreError: 40507:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 720.296069] Lustre: server umount lustre-MDT0000 complete [ 721.796185] LustreError: 38236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 721.802724] LustreError: 38236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 721.957350] LustreError: 38222:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762167101 with bad export cookie 4355908405878380178 [ 721.960350] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 721.963343] LustreError: 38222:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 724.447876] LustreError: 38236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 724.452484] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 724.462648] Lustre: Skipped 1 previous similar message [ 724.465486] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 726.496707] LustreError: 38236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 731.616426] LustreError: 38237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 731.616463] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 731.623632] LustreError: 38237:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 731.627223] Lustre: Skipped 2 previous similar messages [ 736.223214] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 736.266773] LustreError: 40708:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 736.269858] LustreError: 40708:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 736.343323] Lustre: server umount lustre-MDT0001 complete [ 754.419245] LustreError: 40935:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 754.421915] LustreError: 40935:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 754.476729] Lustre: server umount lustre-OST0000 complete [ 760.776078] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 761.997151] Key type lgssc unregistered [ 762.133484] LNet: 41771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.136663] LNetError: 41771:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.145487] LNet: Removed LNI 192.168.202.137@tcp [ 762.498772] Key type .llcrypt unregistered [ 762.499883] Key type ._llcrypt unregistered [ 773.240393] Key type ._llcrypt registered [ 773.241756] Key type .llcrypt registered [ 773.289818] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 773.733503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 773.749166] alg: No test for adler32 (adler32-zlib) [ 774.644422] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 774.744375] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 776.328658] Key type lgssc registered [ 776.791834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 779.975987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 779.984526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 781.198967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 782.704833] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 786.278350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 786.455816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 787.847325] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 789.768281] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 791.045739] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 793.626238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 793.761828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 795.891374] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 797.892239] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 806.057750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 811.491282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 811.491721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 811.493803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 811.496908] Lustre: Skipped 1 previous similar message [ 813.453933] Lustre: lustre-OST0000: Not available for connect from 192.168.202.37@tcp (stopping) [ 813.458551] Lustre: Skipped 1 previous similar message [ 816.609641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 821.729602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 821.732907] Lustre: Skipped 2 previous similar messages [ 825.311175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 825.336035] LustreError: 45409:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 825.385191] Lustre: server umount lustre-OST0000 complete [ 838.776117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.777279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 838.780839] Lustre: Skipped 1 previous similar message [ 838.784825] Lustre: Skipped 3 previous similar messages [ 844.932932] LustreError: 45789:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 844.936733] LustreError: 45789:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 844.987285] Lustre: server umount lustre-MDT0000 complete [ 846.480311] LustreError: 43125:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762167226 with bad export cookie 3655182622169706890 [ 846.483044] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 846.485026] LustreError: 43125:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 852.607982] LustreError: 45990:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 852.609967] LustreError: 45990:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 852.668529] Lustre: server umount lustre-MDT0001 complete [ 856.770309] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 857.941850] Key type lgssc unregistered [ 858.081559] LNet: 46521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.085557] LNetError: 46521:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.093635] LNet: Removed LNI 192.168.202.137@tcp [ 858.446764] Key type .llcrypt unregistered [ 858.448306] Key type ._llcrypt unregistered [ 868.722631] Key type ._llcrypt registered [ 868.724018] Key type .llcrypt registered [ 868.779712] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 869.255466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 869.267954] alg: No test for adler32 (adler32-zlib) [ 870.138838] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 870.241726] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 871.831475] Key type lgssc registered [ 872.305624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 875.358450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 875.368165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 876.576503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 878.045561] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 881.241141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 881.376641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 882.758450] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 884.723526] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 886.060663] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 888.558796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 888.698797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 890.784462] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 892.697558] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 901.987273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 905.226100] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 906.493205] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 917.292111] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-11-03 5h54m55s [ 917.871038] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 922.594156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.594478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 922.595393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 922.601074] Lustre: Skipped 2 previous similar messages [ 923.534440] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.37@tcp (stopping) [ 923.537640] Lustre: Skipped 2 previous similar messages [ 924.546990] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.37@tcp (stopping) [ 924.550556] Lustre: Skipped 3 previous similar messages [ 924.868710] LustreError: 50486:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 924.902500] Lustre: server umount lustre-MDT0000 complete [ 926.166160] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 928.815501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 928.876688] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 928.977989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.007280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 930.021915] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 930.413397] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 931.729424] Lustre: DEBUG MARKER: Wait for df (35981) ... [ 932.261433] Lustre: DEBUG MARKER: done [ 934.368380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 934.371559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.376665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 935.345099] LustreError: 51533:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 935.347540] LustreError: 51533:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 935.398142] Lustre: server umount lustre-OST0000 complete [ 938.944620] LustreError: 51733:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 938.947272] LustreError: 51733:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 938.983177] Lustre: server umount lustre-MDT0000 complete [ 940.414324] LustreError: 47876:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762167320 with bad export cookie 16355548513772309928 [ 940.417530] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 940.419053] LustreError: 47876:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 950.754643] LustreError: 51938:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 950.757670] LustreError: 51938:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 950.807320] Lustre: server umount lustre-MDT0001 complete [ 954.492450] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 955.576329] Key type lgssc unregistered [ 955.705390] LNet: 52470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.708892] LNetError: 52470:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.717791] LNet: Removed LNI 192.168.202.137@tcp [ 956.035595] Key type .llcrypt unregistered [ 956.036955] Key type ._llcrypt unregistered [ 968.237239] Key type ._llcrypt registered [ 968.238757] Key type .llcrypt registered [ 968.281831] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 974.870285] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 975.237729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 975.249461] alg: No test for adler32 (adler32-zlib) [ 976.110023] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 976.205589] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 977.799226] Key type lgssc registered [ 978.236020] Lustre: Echo OBD driver; http://www.lustre.org/ [ 981.472980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 983.866368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 986.085839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 988.319320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 990.763469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 990.787422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 990.794714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 991.884558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 991.897606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 991.937449] Lustre: lustre-MDT0000: new disk, initializing [ 991.975210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.983213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 993.418781] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 997.899840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 997.925413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 997.965552] Lustre: 56394:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 997.989978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 997.992426] Lustre: Skipped 1 previous similar message [ 998.029150] Lustre: lustre-MDT0001: new disk, initializing [ 998.065413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 998.080353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 998.083985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 999.448480] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1002.679490] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1003.924788] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1006.430297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1006.462982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1006.568488] Lustre: lustre-OST0000: new disk, initializing [ 1006.570911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1006.601264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.357078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1007.360783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1007.395155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1008.688445] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1012.094941] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1013.641256] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1013.736145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1015.211885] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1015.295829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1017.826581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.826621] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1017.832278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1017.837228] Lustre: Skipped 1 previous similar message [ 1018.036283] LustreError: 59329:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1018.087876] Lustre: server umount lustre-OST0000 complete [ 1022.945909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1022.946482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1022.950427] Lustre: Skipped 2 previous similar messages [ 1025.728067] LustreError: 59530:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1025.731343] LustreError: 59530:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1025.783873] Lustre: server umount lustre-MDT0000 complete [ 1027.351507] LustreError: 56384:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762167406 with bad export cookie 4894976488595143664 [ 1027.354800] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1027.356649] LustreError: 56384:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1027.421634] LustreError: 59731:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1027.425950] LustreError: 59731:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1027.496558] Lustre: server umount lustre-MDT0001 complete [ 1031.083214] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 05:56:49 (1762167409) [ 1033.924928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1034.156951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.706532] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1039.329452] LustreError: 60401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.336348] LustreError: 60401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1039.402043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1039.591750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1041.238900] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1043.572400] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1045.038589] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1047.784298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1047.920922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1050.064808] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1052.147280] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1065.279976] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-11-03 5h57m24s [ 1065.835880] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1068.372471] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1069.245927] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1069.261819] LustreError: 62556:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1089.424168] Lustre: lustre-MDT0000: Client 8636a467-7ca4-4027-8283-12ae469b59b1 (at 192.168.202.37@tcp) reconnecting [ 1109.303075] LustreError: 62556:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1110.102442] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1110.981589] Lustre: DEBUG MARKER: done [ 1115.105711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1115.108646] LustreError: Skipped 1 previous similar message [ 1115.110506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.116318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1115.118415] Lustre: Skipped 1 previous similar message [ 1118.702083] LustreError: 63424:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1118.703997] LustreError: 63424:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1118.745134] Lustre: server umount lustre-OST0000 complete [ 1120.736217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1120.736853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1120.741639] Lustre: Skipped 2 previous similar messages [ 1120.743838] Lustre: Skipped 1 previous similar message [ 1126.208701] LustreError: 63625:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1126.211672] LustreError: 63625:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1126.249573] Lustre: server umount lustre-MDT0000 complete [ 1126.816788] LustreError: 61088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1126.822561] LustreError: 61088:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1127.667291] LustreError: 60380:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762167507 with bad export cookie 4894976488595144532 [ 1127.669692] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1127.672209] LustreError: 60380:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1127.796550] Lustre: server umount lustre-MDT0001 complete [ 1131.762595] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1132.921750] Key type lgssc unregistered [ 1133.062502] LNet: 64358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.066720] LNetError: 64358:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1133.075452] LNet: Removed LNI 192.168.202.137@tcp [ 1133.423343] Key type .llcrypt unregistered [ 1133.424921] Key type ._llcrypt unregistered [ 1146.185131] Key type ._llcrypt registered [ 1146.186698] Key type .llcrypt registered [ 1146.234310] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 1152.953861] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1153.453455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1153.471786] alg: No test for adler32 (adler32-zlib) [ 1154.347095] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 1154.453736] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1156.055290] Key type lgssc registered [ 1156.501667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1160.068426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1162.520534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1164.723628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1166.983103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1169.419283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1169.446470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1169.451987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1170.541081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1170.554213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1170.595709] Lustre: lustre-MDT0000: new disk, initializing [ 1170.624135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1170.632106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1172.108693] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1176.636963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1176.663758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1176.706963] Lustre: 68282:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1176.740894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1176.743257] Lustre: Skipped 1 previous similar message [ 1176.787269] Lustre: lustre-MDT0001: new disk, initializing [ 1176.828090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1176.843823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1176.848680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1178.315677] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1181.780432] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1183.080503] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1185.685808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1185.709616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1185.815207] Lustre: lustre-OST0000: new disk, initializing [ 1185.817764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1185.856485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1187.956591] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1188.089909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1188.093505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1188.121815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1191.242584] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1192.834242] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1192.929643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1194.430935] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1194.523597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1198.560626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1198.562432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1198.563384] LustreError: Skipped 1 previous similar message [ 1198.564062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1198.568060] Lustre: Skipped 1 previous similar message [ 1201.326090] LustreError: 71217:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1201.367858] Lustre: server umount lustre-OST0000 complete [ 1203.680563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1203.681538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1203.685215] Lustre: Skipped 1 previous similar message [ 1203.688916] Lustre: Skipped 2 previous similar messages [ 1208.801501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1208.805737] Lustre: Skipped 1 previous similar message [ 1208.963069] LustreError: 71419:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1208.965976] LustreError: 71419:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1209.001776] Lustre: server umount lustre-MDT0000 complete [ 1210.426208] LustreError: 69061:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762167590 with bad export cookie 14448676724541371940 [ 1210.428776] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1210.431057] LustreError: 69061:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1210.504324] LustreError: 71621:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1210.507171] LustreError: 71621:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1210.566387] Lustre: server umount lustre-MDT0001 complete [ 1213.512076] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 05:59:52 (1762167592) [ 1214.051353] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1214.610726] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 05:59:53 (1762167593) [ 1218.424699] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1219.886423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1220.036434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1220.363484] Lustre: Failing over lustre-MDT0000 [ 1220.365068] LustreError: 72696:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1220.369347] LustreError: 72696:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1220.372222] LustreError: 72696:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1220.433789] LustreError: 72747:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1220.435726] LustreError: 72747:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1220.480735] Lustre: server umount lustre-MDT0000 complete [ 1222.647075] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 06:00:01 (1762167601) [ 1224.937593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1225.075882] LustreError: 73316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1225.082914] LustreError: 73316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1225.088835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.358362] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1229.028281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1230.375668] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1232.141324] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1233.359318] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1235.766646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1235.887238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1235.890303] Lustre: Skipped 1 previous similar message [ 1237.983733] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1239.954365] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1257.980470] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1260.000147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1260.003785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.009766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1265.853156] LustreError: 75684:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1265.855945] LustreError: 75684:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1265.888596] Lustre: server umount lustre-MDT0000 complete [ 1266.656936] LustreError: 73315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.662376] LustreError: 73315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1267.177034] LustreError: 74034:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762167646 with bad export cookie 14448676724541373081 [ 1267.180037] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1267.180709] LustreError: 74034:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1267.287688] Lustre: server umount lustre-MDT0001 complete [ 1268.460435] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1270.089377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1273.093733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.895763] LustreError: 66495:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e6c0e6b0 x1847766584193024/t0(0) o250->MGC192.168.202.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1277.002767] LustreError: 74923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.003486] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.015027] Lustre: Skipped 2 previous similar messages [ 1277.038902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1277.062887] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1278.288952] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1281.069137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1282.478078] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1284.245936] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1285.431138] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1286.628582] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1287.647169] Lustre: 66496:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762167651/real 1762167651] req@0000000045519186 x1847766584192640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762167667 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1288.671155] Lustre: 66497:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762167652/real 1762167652] req@000000006fba2577 x1847766584192768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762167668 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1291.748678] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1293.151151] Lustre: 66498:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762167656/real 1762167656] req@000000006b9048cf x1847766584192896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762167672 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1296.868195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.870760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1296.871443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1296.871454] Lustre: Skipped 3 previous similar messages [ 1296.873762] Lustre: Skipped 2 previous similar messages [ 1298.244071] LustreError: 78160:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1298.246517] LustreError: 78160:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1298.274241] Lustre: server umount lustre-MDT0000 complete [ 1299.570135] LustreError: 78161:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762167679 with bad export cookie 14448676724541374971 [ 1299.573966] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1299.574866] LustreError: 78161:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1299.684462] Lustre: server umount lustre-MDT0001 complete [ 1301.926636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1305.110662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1310.111603] LustreError: 66495:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007ac4aa33 x1847766584212608/t0(0) o250->MGC192.168.202.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1310.216938] LustreError: 74922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1310.224189] LustreError: 74922:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1310.242921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1310.246270] Lustre: Skipped 1 previous similar message [ 1310.265140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1311.501595] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1314.346632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1314.440428] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1315.753877] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1316.517908] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1317.535112] Lustre: 66497:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762167681/real 1762167681] req@00000000e0ca9b91 x1847766584212224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762167697 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1317.575970] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1318.885238] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1321.956241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1321.958267] Lustre: Skipped 4 previous similar messages [ 1322.975119] Lustre: 66499:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762167686/real 1762167686] req@000000002e53ed43 x1847766584212480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762167702 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1322.983067] Lustre: 66499:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1326.718498] LustreError: 80614:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1326.720538] LustreError: 80614:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1326.748881] Lustre: server umount lustre-MDT0000 complete [ 1327.073890] LustreError: 74923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1327.080402] LustreError: 74923:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1328.020166] LustreError: 73295:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762167707 with bad export cookie 14448676724541376343 [ 1328.022648] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1328.023031] LustreError: 73295:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1328.127494] Lustre: server umount lustre-MDT0001 complete [ 1330.212060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1331.295225] Lustre: DEBUG MARKER: files compared the same [ 1342.234048] Lustre: server umount lustre-OST0000 complete [ 1347.504510] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1348.611267] Key type lgssc unregistered [ 1348.749403] LNet: 82246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1348.752979] LNetError: 82246:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1348.762381] LNet: Removed LNI 192.168.202.137@tcp [ 1349.066680] Key type .llcrypt unregistered [ 1349.068185] Key type ._llcrypt unregistered [ 1357.621068] Key type ._llcrypt registered [ 1357.622082] Key type .llcrypt registered [ 1357.657679] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1357.971574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1357.995071] alg: No test for adler32 (adler32-zlib) [ 1358.852433] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 1358.944065] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1360.527167] Key type lgssc registered [ 1360.859946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1363.459035] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1363.467265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.628686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1365.926680] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug +malloc all [ 1368.701020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1368.797553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1370.012873] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug +malloc all [ 1371.623582] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1372.682140] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1374.859440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1374.949120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.970276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1376.742986] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug +malloc all [ 1378.515794] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1384.417549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1384.417625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1384.425201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1384.427564] Lustre: Skipped 1 previous similar message [ 1387.949977] LustreError: 85859:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1387.983875] Lustre: server umount lustre-OST0000 complete [ 1389.535721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.536158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1389.537259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1389.540781] Lustre: Skipped 2 previous similar messages [ 1394.655451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1394.657052] Lustre: Skipped 2 previous similar messages [ 1395.387585] LustreError: 86066:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1395.390079] LustreError: 86066:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1395.410637] Lustre: server umount lustre-MDT0000 complete [ 1396.655589] LustreError: 86067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762167776 with bad export cookie 14371768959488130854 [ 1396.656915] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1396.658664] LustreError: 86067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1396.722701] LustreError: 86269:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1396.725113] LustreError: 86269:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1396.768661] Lustre: server umount lustre-MDT0001 complete [ 1400.274215] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1401.217554] Key type lgssc unregistered [ 1401.347536] LNet: 86800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.349746] LNetError: 86800:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.358307] LNet: Removed LNI 192.168.202.137@tcp [ 1401.634160] Key type .llcrypt unregistered [ 1401.635670] Key type ._llcrypt unregistered [ 1409.981499] Key type ._llcrypt registered [ 1409.982465] Key type .llcrypt registered [ 1410.015783] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1410.334184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1410.365414] alg: No test for adler32 (adler32-zlib) [ 1411.203184] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 1411.279041] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1412.863113] Key type lgssc registered [ 1413.187411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1415.496087] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1415.502943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1430.943310] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1436.063242] LustreError: 88155:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.137@tcp: failed processing log, type 4: rc = -110 [ 1462.687163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1464.618575] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1466.505225] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1469.127968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1469.163767] Lustre: *** cfs_fail_loc=706, val=0*** [ 1469.295748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1469.314187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1470.609035] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1473.527994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1473.655663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1475.021525] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1476.857273] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1478.147184] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1479.136300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1479.140166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1479.147159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1479.648046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1479.651457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1479.658481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1483.233675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1483.236093] Lustre: Skipped 1 previous similar message [ 1484.973545] LustreError: 90480:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1485.010908] Lustre: server umount lustre-OST0000 complete [ 1488.353772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1488.359215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1488.361854] Lustre: Skipped 1 previous similar message [ 1492.474451] LustreError: 90683:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1492.476228] LustreError: 90683:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1492.503108] Lustre: server umount lustre-MDT0000 complete [ 1493.471944] LustreError: 89132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1493.478132] LustreError: 89132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1493.772121] LustreError: 88164:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762167873 with bad export cookie 9775174585860186268 [ 1493.775521] LustreError: 88164:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1493.776087] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1493.842892] LustreError: 90886:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1493.845681] LustreError: 90886:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1493.901308] Lustre: server umount lustre-MDT0001 complete [ 1497.447815] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1498.595729] Key type lgssc unregistered [ 1498.747635] LNet: 91418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1498.751083] LNetError: 91418:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1498.763794] LNet: Removed LNI 192.168.202.137@tcp [ 1499.119679] Key type .llcrypt unregistered [ 1499.121102] Key type ._llcrypt unregistered [ 1508.071751] Key type ._llcrypt registered [ 1508.072805] Key type .llcrypt registered [ 1508.108914] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1508.461105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1508.509205] alg: No test for adler32 (adler32-zlib) [ 1509.352754] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 1509.432964] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1511.015167] Key type lgssc registered [ 1511.401288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1513.950812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1513.957076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1516.154101] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1518.531913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1518.693366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1519.882277] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1522.513812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1522.617071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1524.433953] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1527.086221] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1527.090886] LustreError: 92150:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009a27eb1b x1847766956382976/t0(0) o250->MGC192.168.202.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1527.096497] LustreError: 93465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1527.097564] Lustre: MGS: Client 4ba7c1f9-bb07-4eed-8802-3918f4dd8820 (at 0@lo) reconnecting [ 1527.101672] LustreError: 93465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 1527.198482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1527.224264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1528.547042] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1537.504868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1537.507421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1537.511434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1539.553176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1539.555028] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1539.555090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1542.445110] LustreError: 95423:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1542.482427] Lustre: server umount lustre-OST0000 complete [ 1544.674291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1544.674548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1544.684666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1544.687896] Lustre: Skipped 2 previous similar messages [ 1549.792707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1549.795814] Lustre: Skipped 1 previous similar message [ 1549.889561] LustreError: 95625:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1549.892513] LustreError: 95625:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1549.901263] Lustre: server umount lustre-MDT0000 complete [ 1551.299073] LustreError: 95827:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1551.301428] LustreError: 95827:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1551.363579] Lustre: server umount lustre-MDT0001 complete [ 1552.751512] Lustre: server umount MGS complete [ 1556.636246] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 06:05:35 (1762167935) [ 1566.005522] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 1569.079034] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1572.399705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1574.355128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1576.179503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1578.150594] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1580.141551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1580.163603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1581.529947] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1585.485737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1585.510987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1585.605022] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1585.627167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1585.670971] Lustre: lustre-MDT0001: new disk, initializing [ 1585.694074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1585.706446] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1585.710428] Lustre: 100533:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1585.713501] LustreError: 100533:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1587.016530] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1591.000859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1591.025974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1591.117130] Lustre: lustre-OST0000: new disk, initializing [ 1591.118736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1591.140345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1592.934927] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1595.864061] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1597.919813] Lustre: 102036:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000008b321e03 x1847766956408704/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1762167993 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1597.968202] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1597.973132] Lustre: MGS: Client 4dda7537-34da-47c8-a383-40135399c081 (at 0@lo) reconnecting [ 1597.984935] Lustre: 99819:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1597.991557] LustreError: 99819:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1598.002889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1598.041787] Lustre: lustre-MDT0000: new disk, initializing [ 1598.089782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1599.135994] Lustre: 102247:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@00000000cf4ea8e0 x1847766956413952/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1762167994 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1599.141117] Lustre: 102247:0:(client.c:1627:after_reply()) Skipped 1 previous similar message [ 1599.279300] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1599.967627] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1604.194254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1604.197890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1604.201619] Lustre: Skipped 1 previous similar message [ 1604.205056] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1623.522140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1623.524555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1623.527022] Lustre: Skipped 2 previous similar messages [ 1633.761427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1633.763963] Lustre: Skipped 4 previous similar messages [ 1637.343095] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1637.361078] LustreError: 102769:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1637.363683] LustreError: 102769:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1637.396171] Lustre: server umount lustre-OST0000 complete [ 1638.881548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1638.881607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1638.885469] Lustre: Skipped 1 previous similar message [ 1644.799027] LustreError: 102972:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1644.801507] LustreError: 102972:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1644.811078] Lustre: server umount lustre-MDT0000 complete [ 1652.530180] Lustre: server umount lustre-MDT0001 complete [ 1656.721447] LustreError: 103376:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1656.730894] LustreError: 103376:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1656.822714] Lustre: server umount MGS complete [ 1660.598495] hrtimer: interrupt took 2620918 ns [ 1669.463067] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 06:07:27 (1762168047) [ 1687.887166] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1690.535965] Key type lgssc unregistered [ 1690.782938] LNet: 104712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.793115] LNetError: 104712:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.809105] LNet: Removed LNI 192.168.202.137@tcp [ 1691.591120] Key type .llcrypt unregistered [ 1691.592704] Key type ._llcrypt unregistered [ 1702.318932] Key type ._llcrypt registered [ 1702.320345] Key type .llcrypt registered [ 1702.383467] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1702.907124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1702.925426] alg: No test for adler32 (adler32-zlib) [ 1703.892729] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 1704.022332] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1705.623170] Key type lgssc registered [ 1706.120812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1709.570196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1709.579762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1710.808299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1712.406204] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1715.837861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1716.007924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1717.813441] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1720.089373] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1721.616817] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1724.683150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1724.855439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1727.299734] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1729.319706] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1736.104799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1746.401674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.401930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1746.403066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1746.406946] Lustre: Skipped 1 previous similar message [ 1748.727369] LustreError: 108158:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1748.767803] Lustre: server umount lustre-OST0000 complete [ 1751.521659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.523263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1751.527241] Lustre: Skipped 1 previous similar message [ 1751.532628] Lustre: Skipped 2 previous similar messages [ 1756.640523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1756.642473] Lustre: Skipped 1 previous similar message [ 1756.676326] LustreError: 108360:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1756.678574] LustreError: 108360:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1756.711559] Lustre: server umount lustre-MDT0000 complete [ 1758.387307] LustreError: 105872:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762168138 with bad export cookie 3419927088098679087 [ 1758.391593] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1758.392607] LustreError: 105872:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1758.469256] LustreError: 108561:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1758.471808] LustreError: 108561:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1758.541359] Lustre: server umount lustre-MDT0001 complete [ 1763.095159] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1764.470423] Key type lgssc unregistered [ 1764.624050] LNet: 109092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.627865] LNetError: 109092:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.640127] LNet: Removed LNI 192.168.202.137@tcp [ 1764.993940] Key type .llcrypt unregistered [ 1764.995800] Key type ._llcrypt unregistered [ 1772.541612] Key type ._llcrypt registered [ 1772.543121] Key type .llcrypt registered [ 1772.591621] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1773.051434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1773.063930] alg: No test for adler32 (adler32-zlib) [ 1773.940946] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 1774.042765] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1775.647184] Key type lgssc registered [ 1776.172555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1779.221307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1779.221964] LustreError: 110133:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1779.224677] Lustre: Skipped 1 previous similar message [ 1779.230492] LustreError: 110132:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1779.233222] LustreError: 110133:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 1779.236477] LustreError: 110133:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1779.241441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.242984] LustreError: 110133:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1780.486592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1783.524623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1783.640792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1785.237298] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1787.031750] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing lsmod [ 1788.114585] LustreError: 111308:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1788.114585] LustreError: 111309:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1788.120571] LustreError: 111308:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 1788.120707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1788.123110] LustreError: 111308:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1788.131306] LustreError: 111308:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1788.256814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.508699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1793.510040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1793.514897] Lustre: Skipped 2 previous similar messages [ 1793.517625] Lustre: Skipped 1 previous similar message [ 1795.531313] LustreError: 111499:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1795.573854] Lustre: server umount lustre-MDT0000 complete [ 1797.269273] LustreError: 111500:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762168176 with bad export cookie 3466375010778639481 [ 1797.272429] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1797.274684] LustreError: 111500:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1797.350498] LustreError: 111701:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1797.352621] LustreError: 111701:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1797.409104] Lustre: server umount lustre-MDT0001 complete [ 1809.150717] LustreError: 111905:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1809.156636] LustreError: 111905:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1809.223918] Lustre: server umount lustre-OST0000 complete [ 1813.544401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1813.743197] LustreError: 112381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1813.753310] LustreError: 112381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1813.775906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1815.610272] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1819.110214] LustreError: 112381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1819.713422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1819.899821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1821.639425] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1824.253268] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1826.288414] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1830.564055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1830.789702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.462415] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1838.094186] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1839.026030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1854.431981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1854.435041] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1854.442447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1855.456720] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1855.461083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1855.461293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1855.470636] Lustre: Skipped 1 previous similar message [ 1857.849777] LustreError: 114647:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1857.852671] LustreError: 114647:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1857.942375] Lustre: server umount lustre-OST0000 complete [ 1860.578675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1860.581520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1860.587574] Lustre: Skipped 1 previous similar message [ 1860.593846] Lustre: Skipped 1 previous similar message [ 1865.696293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1865.699684] Lustre: Skipped 1 previous similar message [ 1874.399850] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1874.459709] LustreError: 114849:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1874.463833] LustreError: 114849:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1874.513900] Lustre: server umount lustre-MDT0000 complete [ 1875.936856] LustreError: 112376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1875.944684] LustreError: 112376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1876.493496] LustreError: 112361:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762168256 with bad export cookie 3466375010778640153 [ 1876.497710] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1876.499593] LustreError: 112361:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1876.659709] Lustre: server umount lustre-MDT0001 complete [ 1883.414839] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1884.965862] Key type lgssc unregistered [ 1885.126821] LNet: 115584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.133316] LNetError: 115584:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1885.142971] LNet: Removed LNI 192.168.202.137@tcp [ 1885.571623] Key type .llcrypt unregistered [ 1885.574911] Key type ._llcrypt unregistered [ 1898.508932] Key type ._llcrypt registered [ 1898.510599] Key type .llcrypt registered [ 1898.557744] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1899.097956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1899.108260] alg: No test for adler32 (adler32-zlib) [ 1900.019850] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 1900.159613] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1901.783420] Key type lgssc registered [ 1902.437332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1906.916565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1906.930699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.240385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1910.608876] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1915.488741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1915.713121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1917.801422] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1920.590140] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1922.344605] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1925.667265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1925.839121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.886701] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1931.356211] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1933.032299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1938.841071] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1939.905490] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1942.497168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1942.501073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1942.508136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1943.523070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1943.523469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1943.524167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.541403] Lustre: Skipped 1 previous similar message [ 1947.643430] LustreError: 119315:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1947.735790] Lustre: server umount lustre-OST0000 complete [ 1951.714521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1951.717445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.724824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1953.762760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1953.763377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.882060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1958.884911] Lustre: Skipped 2 previous similar messages [ 1964.000864] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1964.042338] LustreError: 119521:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1964.044321] LustreError: 119521:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1964.076854] Lustre: server umount lustre-MDT0000 complete [ 1966.008752] LustreError: 116951:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762168345 with bad export cookie 3401642653309909025 [ 1966.011615] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1966.014666] LustreError: 116951:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1966.105282] LustreError: 119723:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1966.108244] LustreError: 119723:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1966.216938] Lustre: server umount lustre-MDT0001 complete [ 1972.151216] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1973.973541] Key type lgssc unregistered [ 1974.162833] LNet: 120254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1974.169817] LNetError: 120254:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1975.205644] LNet: Removed LNI 192.168.202.137@tcp [ 1975.735232] Key type .llcrypt unregistered [ 1975.738651] Key type ._llcrypt unregistered [ 1984.827121] Key type ._llcrypt registered [ 1984.828672] Key type .llcrypt registered [ 1984.879323] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1985.259652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1985.276595] alg: No test for adler32 (adler32-zlib) [ 1986.233189] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 1986.366375] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1987.999174] Key type lgssc registered [ 1988.560979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1993.314447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1993.332066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1994.627346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1996.795452] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2002.006770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2002.206115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2004.397591] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2007.800955] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2010.180909] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2014.585571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2014.833876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2018.263721] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2019.943106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2021.171066] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2030.565424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2030.568386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2030.575054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2032.609445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2032.609544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2032.616290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2035.378139] LustreError: 123683:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2035.429710] Lustre: server umount lustre-OST0000 complete [ 2037.729138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.729758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2037.734633] Lustre: Skipped 1 previous similar message [ 2037.737171] Lustre: Skipped 1 previous similar message [ 2042.848359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2042.851228] Lustre: Skipped 2 previous similar messages [ 2043.136272] LustreError: 123884:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2043.139144] LustreError: 123884:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2043.171628] Lustre: server umount lustre-MDT0000 complete [ 2044.812169] LustreError: 121418:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762168424 with bad export cookie 14402186324901084925 [ 2044.815305] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2044.816731] LustreError: 121418:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2044.888173] LustreError: 124085:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2044.890926] LustreError: 124085:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2044.959061] Lustre: server umount lustre-MDT0001 complete [ 2049.778127] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2051.181854] Key type lgssc unregistered [ 2051.341649] LNet: 124617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2051.346890] LNetError: 124617:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2051.357489] LNet: Removed LNI 192.168.202.137@tcp [ 2051.775475] Key type .llcrypt unregistered [ 2051.777230] Key type ._llcrypt unregistered [ 2063.414732] Key type ._llcrypt registered [ 2063.416261] Key type .llcrypt registered [ 2063.463161] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2063.872916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2063.924731] alg: No test for adler32 (adler32-zlib) [ 2064.813549] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 2064.922839] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2066.527176] Key type lgssc registered [ 2067.010931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2070.204978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2070.212952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2071.404576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.976153] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2076.344634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2076.525315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2078.033211] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2080.269422] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2081.762456] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2084.437766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2084.609980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2086.873772] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2089.083489] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2096.936123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2102.414431] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2104.072056] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2104.075111] Lustre: Skipped 2 previous similar messages [ 2105.960868] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2105.963872] Lustre: Skipped 2 previous similar messages [ 2107.911719] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2107.922290] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2111.580319] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.37@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2111.585249] Lustre: Skipped 2 previous similar messages [ 2118.155033] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.37@tcp 192.168.0.[2,10]@tcp [ 2120.471929] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.37/24@tcp in log lustre-MDT0000 [ 2120.475209] Lustre: Skipped 2 previous similar messages [ 2124.810945] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.37/24@tcp [ 2124.813421] Lustre: Skipped 1 previous similar message [ 2127.841758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2127.843395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.849486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2127.851840] Lustre: Skipped 1 previous similar message [ 2132.784916] LustreError: 129756:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2132.829409] Lustre: server umount lustre-OST0000 complete [ 2138.080827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.080988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2138.081766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2138.081772] Lustre: Skipped 1 previous similar message [ 2138.086211] Lustre: Skipped 1 previous similar message [ 2138.089823] LustreError: Skipped 5 previous similar messages [ 2140.483410] LustreError: 129958:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2140.485965] LustreError: 129958:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2140.515704] Lustre: server umount lustre-MDT0000 complete [ 2142.174595] LustreError: 129009:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762168521 with bad export cookie 2878486392338786198 [ 2142.175352] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2142.180104] LustreError: 129009:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2142.257992] LustreError: 130161:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2142.261060] LustreError: 130161:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2142.320135] Lustre: server umount lustre-MDT0001 complete [ 2146.390580] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2147.652454] Key type lgssc unregistered [ 2147.803486] LNet: 130692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2147.806433] LNetError: 130692:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2147.816372] LNet: Removed LNI 192.168.202.137@tcp [ 2148.169906] Key type .llcrypt unregistered [ 2148.171344] Key type ._llcrypt unregistered [ 2160.540710] Key type ._llcrypt registered [ 2160.542123] Key type .llcrypt registered [ 2160.591491] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2161.091414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2161.106363] alg: No test for adler32 (adler32-zlib) [ 2161.990807] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 2162.097241] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2163.703647] Key type lgssc registered [ 2164.337949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2168.080501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2168.097670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2169.375093] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2169.391083] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.37/24@tcp [ 2169.399398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.237246] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2175.044992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2175.200360] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2175.213461] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.37/24@tcp [ 2175.242350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2177.020587] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2179.367539] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2181.044829] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2184.235891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2184.415443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2187.075408] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2189.319975] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2194.664232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2200.036797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.037230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2200.043044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2200.047486] Lustre: Skipped 1 previous similar message [ 2200.367059] LustreError: 134514:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2200.422850] Lustre: server umount lustre-OST0000 complete [ 2204.039586] LustreError: 134717:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2204.042143] LustreError: 134717:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2204.075381] Lustre: server umount lustre-MDT0000 complete [ 2205.153428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2205.155942] LustreError: 132213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.161041] Lustre: Skipped 2 previous similar messages [ 2205.169904] LustreError: 132213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2205.675397] LustreError: 134718:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762168585 with bad export cookie 8027976527938308440 [ 2205.678735] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2205.678873] LustreError: 134718:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2205.753023] LustreError: 134918:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2205.755122] LustreError: 134918:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2205.813953] Lustre: server umount lustre-MDT0001 complete [ 2209.698869] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2210.866338] Key type lgssc unregistered [ 2211.027926] LNet: 135449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2211.030747] LNetError: 135449:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2211.040373] LNet: Removed LNI 192.168.202.137@tcp [ 2211.401386] Key type .llcrypt unregistered [ 2211.403694] Key type ._llcrypt unregistered [ 2223.099768] Key type ._llcrypt registered [ 2223.101577] Key type .llcrypt registered [ 2223.150939] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 2229.553567] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2230.000400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2230.008215] alg: No test for adler32 (adler32-zlib) [ 2230.885704] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 2230.993751] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2232.583238] Key type lgssc registered [ 2233.097801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2236.937230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2239.579772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2242.081507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2244.870639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2247.840378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2247.871250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2247.880583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.006803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2249.026549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2249.071755] Lustre: lustre-MDT0000: new disk, initializing [ 2249.132944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2249.149162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2250.926647] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2255.941654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2255.969914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2256.009961] Lustre: 138810:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2256.038553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2256.040865] Lustre: Skipped 1 previous similar message [ 2256.076091] Lustre: lustre-MDT0001: new disk, initializing [ 2256.116037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2256.133412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2256.137722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2257.692799] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2261.273915] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2262.750461] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2265.476254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2265.506587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2265.617111] Lustre: lustre-OST0000: new disk, initializing [ 2265.619635] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2265.655590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2267.834613] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2271.291068] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2272.786628] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2274.307267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2274.311260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2274.347899] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2274.928703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2276.333629] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2276.409514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2279.394099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2279.394186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.396597] LustreError: Skipped 1 previous similar message [ 2279.401349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2279.405208] Lustre: Skipped 1 previous similar message [ 2283.120136] LustreError: 141766:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2283.171423] Lustre: server umount lustre-OST0000 complete [ 2287.072160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2287.075841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2287.081371] Lustre: Skipped 1 previous similar message [ 2287.084036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2289.632214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2289.632692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2290.688697] LustreError: 141967:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2290.691186] LustreError: 141967:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2290.718096] Lustre: server umount lustre-MDT0000 complete [ 2292.056033] LustreError: 138801:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762168671 with bad export cookie 6949439255046475728 [ 2292.058692] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2292.061221] LustreError: 138801:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2292.140645] LustreError: 142168:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2292.142625] LustreError: 142168:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2292.206832] Lustre: server umount lustre-MDT0001 complete [ 2295.311089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2295.498543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2296.935500] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2300.127673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2300.302227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2301.887175] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2304.041257] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2305.476707] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2307.985778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2310.274879] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2312.236541] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2313.830756] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2321.104673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 2322.664375] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2322.751859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2327.794222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2327.832699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2327.905374] Lustre: lustre-OST0001: new disk, initializing [ 2327.907920] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2327.945335] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2327.948452] Lustre: Skipped 1 previous similar message [ 2330.267343] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2334.037888] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2335.757211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2335.761488] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2335.801919] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2336.184373] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2337.811538] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2337.905708] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2339.419924] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99afc6aa1000.ost_server_uuid 50 [ 2339.953395] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99afc6aa1000.ost_server_uuid in FULL state after 0 sec [ 2345.962187] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2345.963162] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2345.965650] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2345.965665] Lustre: Skipped 1 previous similar message [ 2345.967177] Lustre: Skipped 1 previous similar message [ 2345.970620] LustreError: Skipped 1 previous similar message [ 2348.212173] LustreError: 146954:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2348.215834] LustreError: 146954:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2348.277323] Lustre: server umount lustre-OST0001 complete [ 2351.078867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2351.079240] LustreError: 144240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.079891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2351.085509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2351.090069] LustreError: 144240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2351.093092] Lustre: Skipped 2 previous similar messages [ 2356.195489] LustreError: 146868:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.200874] LustreError: 146868:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2361.312970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2361.316175] Lustre: Skipped 3 previous similar messages [ 2361.316601] LustreError: 145975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.326062] LustreError: 145975:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2364.383142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2364.401090] LustreError: 147155:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2364.404039] LustreError: 147155:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2364.445336] Lustre: server umount lustre-OST0000 complete [ 2366.432927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.437647] Lustre: Skipped 1 previous similar message [ 2380.255146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2380.295744] LustreError: 147358:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2380.298134] LustreError: 147358:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2380.330418] Lustre: server umount lustre-MDT0000 complete [ 2381.781748] LustreError: 142624:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762168761 with bad export cookie 6949439255046476603 [ 2381.785072] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2381.787052] LustreError: 142624:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2381.932768] Lustre: server umount lustre-MDT0001 complete [ 2389.362056] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2390.596818] Key type lgssc unregistered [ 2390.736839] LNet: 148549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.741166] LNetError: 148549:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2390.750536] LNet: Removed LNI 192.168.202.137@tcp [ 2391.093319] Key type .llcrypt unregistered [ 2391.094724] Key type ._llcrypt unregistered [ 2403.784415] Key type ._llcrypt registered [ 2403.785697] Key type .llcrypt registered [ 2403.829318] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 2410.897921] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2411.326776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2411.356715] alg: No test for adler32 (adler32-zlib) [ 2412.230397] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 2412.327034] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2413.919251] Key type lgssc registered [ 2414.395678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2417.782880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2420.120671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2422.511542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2424.956780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2427.420842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2427.442391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2427.448677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.561315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2428.581453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2428.623964] Lustre: lustre-MDT0000: new disk, initializing [ 2428.676295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2428.689342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2430.240356] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2434.866259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2434.891940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2434.932406] Lustre: 152471:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2434.961226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2434.963692] Lustre: Skipped 1 previous similar message [ 2435.002617] Lustre: lustre-MDT0001: new disk, initializing [ 2435.043772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2435.059241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2435.064083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2436.612885] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2440.131930] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2441.706343] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2444.397929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2444.430727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2444.547292] Lustre: lustre-OST0000: new disk, initializing [ 2444.549881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2444.589589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2446.209027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2446.213134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2446.255869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2446.737596] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2450.218333] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2451.808690] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2451.901284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2453.531414] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2453.627409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2456.544355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2456.545330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2456.547538] LustreError: Skipped 1 previous similar message [ 2456.552256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2456.555768] Lustre: Skipped 1 previous similar message [ 2460.463112] LustreError: 155407:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2460.516982] Lustre: server umount lustre-OST0000 complete [ 2465.760675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2465.764187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2465.769460] Lustre: Skipped 1 previous similar message [ 2465.773216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2466.785787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2466.785917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2468.355509] LustreError: 155608:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2468.358553] LustreError: 155608:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2468.395067] Lustre: server umount lustre-MDT0000 complete [ 2469.887935] LustreError: 153252:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762168849 with bad export cookie 15039165262990266872 [ 2469.889952] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2469.892990] LustreError: 153252:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2469.967906] LustreError: 155809:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2469.970853] LustreError: 155809:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2470.034065] Lustre: server umount lustre-MDT0001 complete [ 2473.653566] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 06:20:52 (1762168852) [ 2479.889553] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 2483.548984] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2487.406707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2489.857481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2492.351956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2494.736384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2497.347990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2497.372172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2497.463694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2497.478358] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2497.519833] Lustre: lustre-MDT0000: new disk, initializing [ 2497.552697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2497.561615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2498.984965] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2503.636809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2503.661932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2503.700274] Lustre: 158794:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2503.729670] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2503.732120] Lustre: Skipped 1 previous similar message [ 2503.772726] Lustre: lustre-MDT0001: new disk, initializing [ 2503.815496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2503.835971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2503.840341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2505.396052] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2508.957348] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2510.286961] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2512.706202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2512.730251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2512.833852] Lustre: lustre-OST0000: new disk, initializing [ 2512.836082] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2512.866158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2514.785667] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2518.019584] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2519.545886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2519.549858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2519.579903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2525.430403] Lustre: Failing over lustre-OST0000 [ 2525.454794] LustreError: 161236:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2525.456842] LustreError: 161236:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2525.503754] Lustre: server umount lustre-OST0000 complete [ 2538.441359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2538.556475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.561171] Lustre: Skipped 1 previous similar message [ 2538.565618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2538.572134] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2539.913467] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2540.435217] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2541.710894] Lustre: Failing over lustre-MDT0000 [ 2541.782297] LustreError: 162207:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2541.784592] LustreError: 162207:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2541.813171] Lustre: server umount lustre-MDT0000 complete [ 2543.585745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.591362] Lustre: Skipped 2 previous similar messages [ 2545.631120] Lustre: 150686:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762168909/real 1762168909] req@0000000086ea583a x1847767902252800/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1762168925 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2548.704263] LustreError: 159907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.711623] LustreError: 159907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2550.145761] LustreError: 158801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2550.751257] Lustre: 150686:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762168914/real 1762168914] req@00000000552ad113 x1847767902253824/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1762168930 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2550.759766] Lustre: 150686:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2553.824139] LustreError: 159907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.830846] LustreError: 159907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2554.953044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2555.006910] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2555.012652] Lustre: 158796:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.202.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2555.096885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2555.142214] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2555.265807] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2556.623880] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2560.485989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2560.494544] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2560.498533] Lustre: Skipped 1 previous similar message [ 2560.516790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2565.602200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2565.602418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.605505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2565.608630] Lustre: Skipped 1 previous similar message [ 2565.613236] Lustre: Skipped 2 previous similar messages [ 2567.794217] LustreError: 163091:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2567.797134] LustreError: 163091:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2567.839475] Lustre: server umount lustre-OST0000 complete [ 2570.722841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2570.724053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2570.726627] LustreError: Skipped 1 previous similar message [ 2570.729503] Lustre: Skipped 1 previous similar message [ 2575.383614] Lustre: server umount lustre-MDT0000 complete [ 2575.840052] LustreError: 158805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.845161] LustreError: 158805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2576.798869] LustreError: 159575:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762168956 with bad export cookie 15039165262990268909 [ 2576.802342] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2576.804349] LustreError: 159575:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2576.944429] Lustre: server umount lustre-MDT0001 complete [ 2581.108704] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2582.249793] Key type lgssc unregistered [ 2582.375494] LNet: 164026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2582.379232] LNetError: 164026:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2582.389524] LNet: Removed LNI 192.168.202.137@tcp [ 2582.728474] Key type .llcrypt unregistered [ 2582.730118] Key type ._llcrypt unregistered [ 2603.528509] Key type ._llcrypt registered [ 2603.530082] Key type .llcrypt registered [ 2603.573968] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2603.981398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2604.035111] alg: No test for adler32 (adler32-zlib) [ 2604.926993] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 2605.015289] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2606.599223] Key type lgssc registered [ 2606.976477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2609.793045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2609.798458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2610.956642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2612.391764] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2615.847737] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2617.875127] LustreError: 165553:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2617.878979] LustreError: 165553:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2617.882554] LustreError: 165553:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2617.948775] LustreError: 166075:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2618.006100] Lustre: server umount lustre-MDT0000 complete [ 2623.235356] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 2626.958589] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2630.514702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2632.871146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2635.099231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2637.538802] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2640.090758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2640.115656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2640.203988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2640.218356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2640.256282] Lustre: lustre-MDT0000: new disk, initializing [ 2640.281363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2640.289699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2641.729712] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2646.186237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2646.213769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2646.256348] Lustre: 168714:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2646.284678] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2646.287233] Lustre: Skipped 1 previous similar message [ 2646.323804] Lustre: lustre-MDT0001: new disk, initializing [ 2646.358832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2646.372612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2646.376580] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2647.849826] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2651.060310] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2652.338878] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2654.777666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2654.803407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2654.905155] Lustre: lustre-OST0000: new disk, initializing [ 2654.907856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2654.940075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.087834] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2657.976170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2657.980101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2658.011696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2660.320300] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2662.193412] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2662.832690] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 06:24:01 (1762169041) [ 2779.615692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2779.619513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2779.634675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2781.157716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2781.157887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2783.350431] LustreError: 172220:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2783.365904] LustreError: 172220:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2783.476836] Lustre: server umount lustre-MDT0000 complete [ 2786.271757] LustreError: 169500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.281441] LustreError: 169500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2786.416750] LustreError: 169494:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762169166 with bad export cookie 6569673156609380663 [ 2786.418643] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2786.430273] LustreError: 169494:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2791.392257] LustreError: 171221:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2791.392716] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2791.408126] Lustre: Skipped 1 previous similar message [ 2791.410800] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2791.413381] Lustre: Skipped 2 previous similar messages [ 2792.596508] LustreError: 172423:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2792.599216] LustreError: 172423:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2792.678485] Lustre: server umount lustre-MDT0001 complete [ 2801.520404] LustreError: 172626:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2801.522393] LustreError: 172626:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2801.594968] Lustre: server umount lustre-OST0000 complete [ 2807.199299] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 2812.563420] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2818.364272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2822.407751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2825.922455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2829.678358] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2833.990879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2834.043842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2834.188299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2834.210563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2834.257830] Lustre: lustre-MDT0000: new disk, initializing [ 2834.314447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2834.332574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2836.892953] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2843.341994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2843.390770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2843.454887] Lustre: 174965:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2843.501342] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2843.504393] Lustre: Skipped 1 previous similar message [ 2843.575917] Lustre: lustre-MDT0001: new disk, initializing [ 2843.631658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2843.652859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2843.658499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2846.190914] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2851.131674] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2853.380259] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2857.698519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2857.742700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2857.895505] Lustre: lustre-OST0000: new disk, initializing [ 2857.899891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2861.591759] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2866.191356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2866.198745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2866.281314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2866.551740] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2868.781973] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2868.922765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2871.188965] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2871.326959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2876.384252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2876.385980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.389435] LustreError: Skipped 1 previous similar message [ 2876.397321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2878.650949] LustreError: 177908:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2878.655086] LustreError: 177908:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2878.729117] Lustre: server umount lustre-OST0000 complete [ 2881.504924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2881.511815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2881.511823] Lustre: Skipped 1 previous similar message [ 2881.519037] Lustre: Skipped 2 previous similar messages [ 2891.746134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2891.749763] Lustre: Skipped 4 previous similar messages [ 2895.327397] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2895.376967] LustreError: 178109:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2895.380381] LustreError: 178109:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2895.417381] Lustre: server umount lustre-MDT0000 complete [ 2896.864230] LustreError: 174976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2896.870302] LustreError: 174976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2896.892544] LustreError: 175752:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762169276 with bad export cookie 6569673156609539479 [ 2896.895869] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2896.897282] LustreError: 175752:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2897.029403] Lustre: server umount lustre-MDT0001 complete [ 2901.136531] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:27:59 (1762169279) [ 2910.624309] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 2916.079626] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2921.604845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2925.005225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2928.334402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2931.831216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2935.224404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2935.273172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2935.381206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2935.402317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2935.448808] Lustre: lustre-MDT0000: new disk, initializing [ 2935.496440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2935.499915] Lustre: Skipped 1 previous similar message [ 2935.509448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2937.380237] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2943.801079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2943.835991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2943.893849] Lustre: 181300:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2944.044377] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2946.236087] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2950.934471] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2953.135603] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2956.987850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2957.023290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2958.605434] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2958.675083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2960.110455] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2964.381075] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2969.058943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2969.059229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.060111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2969.062779] LustreError: Skipped 1 previous similar message [ 2969.068301] Lustre: Skipped 1 previous similar message [ 2973.566497] LustreError: 183849:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2973.568978] LustreError: 183849:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2973.642419] Lustre: server umount lustre-OST0000 complete [ 2981.881426] Lustre: server umount lustre-MDT0000 complete [ 2983.811903] LustreError: 181291:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762169363 with bad export cookie 6569673156609540361 [ 2983.814819] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2983.818695] LustreError: 181291:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2984.017659] Lustre: server umount lustre-MDT0001 complete [ 2988.266273] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:29:26 (1762169366) [ 2996.447584] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 3001.561186] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3007.086150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3010.590859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3014.154845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3018.223632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3022.225589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3022.267575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3022.382423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3022.401682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3022.404230] Lustre: Skipped 3 previous similar messages [ 3022.445362] Lustre: lustre-MDT0000: new disk, initializing [ 3022.447614] Lustre: Skipped 2 previous similar messages [ 3022.492886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3022.495667] Lustre: Skipped 2 previous similar messages [ 3022.504496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3022.509147] Lustre: Skipped 2 previous similar messages [ 3024.536530] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3030.868485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3030.902803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3030.964845] Lustre: 187240:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3031.122874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3033.281228] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3038.089675] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3040.011214] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3043.525484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3043.556591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3046.504456] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3051.030522] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3053.130388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3063.266453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3063.266803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3063.268229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3063.268240] Lustre: Skipped 3 previous similar messages [ 3063.271293] Lustre: Skipped 3 previous similar messages [ 3065.202181] LustreError: 189783:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3065.208052] LustreError: 189783:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3065.272383] Lustre: server umount lustre-OST0000 complete [ 3073.327525] Lustre: server umount lustre-MDT0000 complete [ 3073.509756] LustreError: 189406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.402949] LustreError: 187232:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762169455 with bad export cookie 6569673156609541446 [ 3075.405082] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3075.407959] LustreError: 187232:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3075.597180] Lustre: server umount lustre-MDT0001 complete [ 3080.522460] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3081.772507] Key type lgssc unregistered [ 3081.937436] LNet: 190718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3081.941232] LNetError: 190718:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3081.952360] LNet: Removed LNI 192.168.202.137@tcp [ 3082.329627] Key type .llcrypt unregistered [ 3082.331649] Key type ._llcrypt unregistered [ 3093.601694] Key type ._llcrypt registered [ 3093.603291] Key type .llcrypt registered [ 3093.648876] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3094.133304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3094.145524] alg: No test for adler32 (adler32-zlib) [ 3095.036340] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 3095.144204] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 3096.743147] Key type lgssc registered [ 3097.215194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3100.831853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3100.842348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3102.099901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3103.999591] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3108.211696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3108.399902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3110.118763] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3112.475881] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3113.993495] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3116.915794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3117.082765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3119.887954] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3122.544663] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3127.334709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3142.624980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3142.625311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.629277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3142.633458] Lustre: Skipped 1 previous similar message [ 3146.540562] LustreError: 194361:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3146.591391] Lustre: server umount lustre-OST0000 complete [ 3149.280346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3149.284148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3149.289466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3149.291763] Lustre: Skipped 1 previous similar message [ 3152.867571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3152.868437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.316206] LustreError: 194564:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3154.319360] LustreError: 194564:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3154.371675] Lustre: server umount lustre-MDT0000 complete [ 3156.187664] LustreError: 192074:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762169535 with bad export cookie 1665638645188926670 [ 3156.191071] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3156.193796] LustreError: 192074:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3156.282411] LustreError: 194766:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3156.286673] LustreError: 194766:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3156.365599] Lustre: server umount lustre-MDT0001 complete [ 3160.895092] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3162.199985] Key type lgssc unregistered [ 3162.346164] LNet: 195297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3162.351271] LNetError: 195297:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3162.362612] LNet: Removed LNI 192.168.202.137@tcp [ 3162.751348] Key type .llcrypt unregistered [ 3162.753294] Key type ._llcrypt unregistered [ 3174.955578] Key type ._llcrypt registered [ 3174.957942] Key type .llcrypt registered [ 3175.015641] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3175.490540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3175.522403] alg: No test for adler32 (adler32-zlib) [ 3176.426456] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 3176.532643] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 3178.127260] Key type lgssc registered [ 3178.621373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3182.228160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3182.243217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3183.495392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3185.313974] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3189.420969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3189.620824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3191.569550] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3194.563227] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3196.688973] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3200.194938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3200.368087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3203.444815] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3205.900150] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3208.551122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3219.937819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3219.938832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.944699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3219.946154] Lustre: Skipped 1 previous similar message [ 3221.682169] LustreError: 198935:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3221.736994] Lustre: server umount lustre-OST0000 complete [ 3224.686420] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff99afd2baa000.ost_server_uuid 50 [ 3240.411353] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3251.313601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3251.321034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3251.323322] Lustre: Skipped 1 previous similar message [ 3256.292651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3256.299512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3256.318271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3257.486345] LustreError: 199304:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3257.488971] LustreError: 199304:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3257.531390] Lustre: server umount lustre-MDT0000 complete [ 3259.548216] LustreError: 196652:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762169639 with bad export cookie 16904500967789581023 [ 3259.549218] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3259.553731] LustreError: 196652:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3259.643876] LustreError: 199507:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3259.646872] LustreError: 199507:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3259.747381] Lustre: server umount lustre-MDT0001 complete [ 3264.076853] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 06:34:02 (1762169642) [ 3267.458506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3267.726412] LustreError: 200177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3267.736821] LustreError: 200177:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3267.779743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3269.933755] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3273.186272] LustreError: 200177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3274.700382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3274.940399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3276.902790] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3279.644907] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3281.622757] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3285.544868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3285.750415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3289.206471] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3292.325928] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3292.906581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3296.026333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3296.084833] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3296.196110] Lustre: lustre-OST0001: new disk, initializing [ 3296.198985] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3296.418113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3296.424821] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3296.503660] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3299.327344] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3303.943838] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3308.513109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3308.518923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.531700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3308.534019] Lustre: Skipped 1 previous similar message [ 3311.414175] LustreError: 203380:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3311.417055] LustreError: 203380:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3311.447727] Lustre: server umount lustre-OST0000 complete [ 3314.563746] LustreError: 201772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.931367] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3315.085944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3317.407267] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3317.528958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3319.688657] LustreError: 201775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3319.698824] LustreError: 201775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3324.806238] LustreError: 201772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3324.815951] LustreError: 201772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3330.532644] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3330.535755] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3330.537876] Lustre: Skipped 2 previous similar messages [ 3330.540732] Lustre: Skipped 3 previous similar messages [ 3334.132256] LustreError: 204083:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3334.139719] LustreError: 204083:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3334.238069] Lustre: server umount lustre-OST0001 complete [ 3336.675291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3336.679378] LustreError: Skipped 1 previous similar message [ 3340.769207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3340.770086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.774284] Lustre: Skipped 2 previous similar messages [ 3340.779618] Lustre: Skipped 1 previous similar message [ 3342.477969] LustreError: 204284:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3342.480124] LustreError: 204284:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3342.540848] Lustre: server umount lustre-MDT0000 complete [ 3344.445428] LustreError: 200895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762169724 with bad export cookie 16904500967789582087 [ 3344.447720] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3344.452230] LustreError: 200895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3344.640736] Lustre: server umount lustre-MDT0001 complete [ 3347.691223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3349.850145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3351.892171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3354.063810] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3357.679148] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 06:35:36 (1762169736) [ 3361.224243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.373306] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3361.401741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3361.504474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3361.508372] Lustre: Skipped 1 previous similar message [ 3363.402615] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3367.762483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3367.816776] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3367.844328] Lustre: 205975:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3369.820898] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3372.687141] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3374.598087] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3377.810208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3377.923337] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3380.580549] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3383.145957] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3383.341444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3386.246515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3386.286458] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3389.277370] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3391.745313] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3393.504458] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3393.504835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3393.507938] LustreError: Skipped 1 previous similar message [ 3393.513878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3393.518242] Lustre: Skipped 2 previous similar messages [ 3399.152674] LustreError: 209172:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3399.156765] LustreError: 209172:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3399.185978] Lustre: server umount lustre-OST0000 complete [ 3402.120417] LustreError: 207605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.135436] LustreError: 207605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3412.450332] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3416.742916] Lustre: server umount lustre-OST0001 complete [ 3424.788141] Lustre: server umount lustre-MDT0000 complete [ 3426.691241] LustreError: 206707:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762169806 with bad export cookie 16904500967789583487 [ 3426.692182] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3426.699279] LustreError: 206707:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3426.859192] Lustre: server umount lustre-MDT0001 complete [ 3429.706867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3431.740304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3433.767968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3435.670113] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3438.867409] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 06:36:57 (1762169817) [ 3446.215624] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 3450.383530] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3455.132839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3458.132191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3460.804988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3463.984761] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3467.604456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3467.637447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3467.737867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3467.755434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3467.797428] Lustre: lustre-MDT0000: new disk, initializing [ 3467.841225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3467.843642] Lustre: Skipped 3 previous similar messages [ 3467.852786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3469.636544] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3474.970553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3475.001471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3475.046134] Lustre: 213600:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3475.079311] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3475.082033] Lustre: Skipped 1 previous similar message [ 3475.122264] Lustre: lustre-MDT0001: new disk, initializing [ 3475.194824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3475.200589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3476.837955] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3480.453315] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3481.876776] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3484.518265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3484.547410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3484.651978] Lustre: lustre-OST0000: new disk, initializing [ 3484.654538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3486.994676] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3489.289047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3489.294220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3489.357127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3490.768420] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3492.709983] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3492.817655] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3494.821397] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3494.935523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3499.489887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3499.490153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3499.491476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3499.491483] Lustre: Skipped 8 previous similar messages [ 3499.494946] Lustre: Skipped 6 previous similar messages [ 3499.497146] LustreError: Skipped 3 previous similar messages [ 3510.239197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3510.263103] LustreError: 216538:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3510.269104] LustreError: 216538:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3510.330543] Lustre: server umount lustre-OST0000 complete [ 3518.506800] Lustre: server umount lustre-MDT0000 complete [ 3519.969549] LustreError: 214997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3519.981048] LustreError: 214997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3520.312535] LustreError: 214381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762169899 with bad export cookie 16904500967789585048 [ 3520.315493] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3520.319441] LustreError: 214381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3520.478685] Lustre: server umount lustre-MDT0001 complete [ 3524.499430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.759961] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3530.836546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3533.001454] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3535.699843] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3537.464453] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3540.992760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3541.168579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3541.174581] Lustre: Skipped 4 previous similar messages [ 3543.851204] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3546.603492] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3548.454071] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3550.642325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3552.511199] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3552.621233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3555.809060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3559.643808] Lustre: server umount lustre-OST0000 complete [ 3562.076466] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3563.784402] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3588.176530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3591.016806] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3593.579878] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3594.296367] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3595.492415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3595.499925] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3595.500927] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3595.511754] Lustre: Skipped 1 previous similar message [ 3600.864970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3600.870887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3600.875498] Lustre: Skipped 5 previous similar messages [ 3600.881521] Lustre: Skipped 10 previous similar messages [ 3601.775171] LustreError: 221810:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3601.784873] LustreError: 221810:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3601.870041] Lustre: server umount lustre-OST0000 complete [ 3618.271541] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3618.349081] Lustre: server umount lustre-MDT0000 complete [ 3619.819288] LustreError: 217397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762169999 with bad export cookie 16904500967789585930 [ 3619.821870] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3619.824179] LustreError: 217397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3619.954516] Lustre: server umount lustre-MDT0001 complete [ 3623.512290] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 06:40:02 (1762170002) [ 3626.266209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3626.454649] LustreError: 222889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3626.464830] LustreError: 222889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3628.165330] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3631.915530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3633.656829] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3635.813812] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3637.299986] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3640.231647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3642.682858] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3644.916508] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3646.654675] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3651.870306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 3653.431714] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3653.526505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3656.229760] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3656.265318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3656.329762] Lustre: lustre-OST0001: new disk, initializing [ 3656.332588] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3658.666966] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3662.424605] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3664.135420] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3666.441677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3666.446038] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3666.487692] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3667.316446] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3669.087741] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3669.190216] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3671.519959] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3671.523812] LustreError: Skipped 2 previous similar messages [ 3676.049393] Lustre: server umount lustre-OST0001 complete [ 3678.627886] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3678.741527] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3680.499523] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3680.599875] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3691.395743] LustreError: 226480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3691.404750] LustreError: 226480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 3705.094909] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3705.228836] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3705.233596] Lustre: Skipped 5 previous similar messages [ 3707.787635] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3709.951470] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3710.439031] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3710.444547] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3710.444704] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3710.601154] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3713.412080] Lustre: server umount lustre-OST0001 complete [ 3728.864398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3728.869945] Lustre: Skipped 7 previous similar messages [ 3728.872286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3728.874855] Lustre: Skipped 10 previous similar messages [ 3732.023528] LustreError: 229029:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3732.026647] LustreError: 229029:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3732.069179] Lustre: server umount lustre-OST0000 complete [ 3739.812053] Lustre: server umount lustre-MDT0000 complete [ 3741.279056] LustreError: 225928:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762170120 with bad export cookie 16904500967789586875 [ 3741.283693] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3741.285016] LustreError: 225928:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3747.455711] Lustre: server umount lustre-MDT0001 complete [ 3749.630939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3751.236542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3752.979422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3754.650323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3757.453445] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 06:42:16 (1762170136) [ 3760.178857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3760.311126] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3760.325509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3762.006842] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3765.757575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3765.810970] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3765.827159] Lustre: 230923:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3767.646862] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3769.801944] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3771.205889] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3773.930418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3774.018783] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3776.350363] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3778.522479] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3782.719499] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3782.755912] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3785.182361] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3787.622173] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3788.373466] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3790.079823] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3793.271263] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3795.009720] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3795.111727] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3796.775903] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99afc9145800.ost_server_uuid 50 [ 3797.466066] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99afc9145800.ost_server_uuid in FULL state after 0 sec [ 3797.930166] Lustre: Permanently deactivating lustre-OST0001 [ 3797.939506] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3798.935748] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3799.240715] Lustre: Permanently reactivating lustre-OST0001 [ 3806.731698] Lustre: server umount lustre-OST0001 complete [ 3814.368249] Lustre: server umount lustre-OST0000 complete [ 3827.168309] LustreError: 234943:0:(osp_dev.c:692:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 3830.239297] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3830.339641] Lustre: server umount lustre-MDT0000 complete [ 3832.146839] LustreError: 232561:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762170211 with bad export cookie 16904500967789588149 [ 3832.152420] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3832.384130] Lustre: server umount lustre-MDT0001 complete [ 3835.042583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3836.705072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3838.517526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3840.102904] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3842.898451] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 06:43:41 (1762170221) [ 3843.353543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3846.276904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3846.419187] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3846.434100] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3846.439083] Lustre: Skipped 5 previous similar messages [ 3848.092306] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3851.568792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3851.628801] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3851.641925] Lustre: 237092:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3853.409877] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3855.516113] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3856.942025] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3859.761076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3859.791453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3859.892761] Lustre: Permanently deactivating lustre-OST0000 [ 3859.898904] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3862.349375] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3863.804713] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3863.807159] Lustre: Skipped 1 previous similar message [ 3864.563926] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3867.363930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3869.925043] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3870.577730] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3872.256582] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3873.004615] Lustre: Permanently reactivating lustre-OST0000 [ 3880.455238] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3880.458971] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3880.464414] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3880.466983] Lustre: Skipped 1 previous similar message [ 3880.484790] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3880.495859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3885.029430] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 3885.033216] LustreError: Skipped 5 previous similar messages [ 3888.708870] Lustre: server umount lustre-OST0001 complete [ 3891.171715] LustreError: 238718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3891.180414] LustreError: 238718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 3896.353719] Lustre: server umount lustre-OST0000 complete [ 3906.231765] Lustre: server umount lustre-MDT0000 complete [ 3907.848316] LustreError: 237083:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762170287 with bad export cookie 16904500967789589871 [ 3907.850842] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3907.853830] LustreError: 237083:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3907.986913] Lustre: server umount lustre-MDT0001 complete [ 3912.184721] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3913.452056] Key type lgssc unregistered [ 3913.607509] LNet: 241484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3913.610897] LNetError: 241484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3913.618372] LNet: Removed LNI 192.168.202.137@tcp [ 3913.982841] Key type .llcrypt unregistered [ 3913.985572] Key type ._llcrypt unregistered [ 3924.084757] Key type ._llcrypt registered [ 3924.086285] Key type .llcrypt registered [ 3924.134867] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3924.645955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3924.655602] alg: No test for adler32 (adler32-zlib) [ 3925.537169] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 3925.638251] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 3927.239174] Key type lgssc registered [ 3927.746410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3929.195271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3932.059780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3932.068537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3933.301591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3935.023876] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3938.498759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3938.530071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3938.581844] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3938.596225] Lustre: 242899:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3938.601770] Lustre: Permanently deactivating lustre-MDT0001 [ 3938.608713] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3938.639826] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3938.644033] LustreError: 242941:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3938.649904] LustreError: 242941:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3938.653944] LustreError: 242941:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 3938.760954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3940.305939] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3942.413838] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3943.820152] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3946.377382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3946.514464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3948.580803] LustreError: 244840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3948.588611] LustreError: 244840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3948.678714] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3950.836919] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3953.623718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3953.634452] LustreError: 244840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3953.734369] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3956.113841] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3957.866550] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3957.867207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3957.874865] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3958.457528] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3960.220367] Lustre: Permanently reactivating lustre-MDT0001 [ 3960.229370] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3960.231789] Lustre: Skipped 2 previous similar messages [ 3964.814244] Lustre: lustre-MDT0001: Client d25df71c-ad41-44a0-bb70-a020b19b354d (at 192.168.202.37@tcp) reconnecting [ 3966.742602] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3967.384105] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3969.456434] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3969.549528] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3971.223840] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3971.323534] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3971.815444] Lustre: 242907:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 3971.821328] Lustre: 242907:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 3971.825139] Lustre: 242907:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 3971.830481] Lustre: 242907:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 3971.834841] Lustre: 242907:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 3971.838712] Lustre: 242907:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 3972.409116] Lustre: Permanently deactivating lustre-MDT0001 [ 3972.417302] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3972.420085] Lustre: Skipped 2 previous similar messages [ 3981.315571] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3985.548572] Lustre: 242906:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 504 < left 979, rollback = 2 [ 3985.552947] Lustre: 242906:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 1 previous similar message [ 3985.556088] Lustre: 242906:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/8, destroy: 0/0/0 [ 3985.559533] Lustre: 242906:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3985.563457] Lustre: 242906:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 6/979/0 [ 3985.566913] Lustre: 242906:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3985.570825] Lustre: 242906:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3985.573749] Lustre: 242906:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3985.577266] Lustre: 242906:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 8/130/4, delete: 0/0/0 [ 3985.580641] Lustre: 242906:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3985.584065] Lustre: 242906:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0 [ 3985.587157] Lustre: 242906:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3988.964317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.966023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3988.970931] Lustre: Skipped 2 previous similar messages [ 3988.973159] Lustre: Skipped 1 previous similar message [ 3989.185942] LustreError: 247023:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3989.217713] Lustre: server umount lustre-MDT0000 complete [ 3990.704719] LustreError: 246012:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762170370 with bad export cookie 3944763711649158630 [ 3990.708103] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3990.709947] LustreError: 246012:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3990.781857] LustreError: 247223:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3990.784286] LustreError: 247223:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3990.844974] Lustre: server umount lustre-MDT0001 complete [ 3998.325854] LustreError: 247424:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3998.327962] LustreError: 247424:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3998.346150] Lustre: server umount lustre-OST0000 complete [ 4002.845461] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 06:46:21 (1762170381) [ 4006.932460] LustreError: 248430:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4006.935067] LustreError: 248430:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4006.975075] Lustre: server umount lustre-OST0001 complete [ 4009.255976] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 4012.844022] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4016.414579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4018.900424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4021.147459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4023.668079] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4026.257798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4026.281457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4026.377757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4026.382249] Lustre: Skipped 2 previous similar messages [ 4026.396285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4026.440349] Lustre: lustre-MDT0000: new disk, initializing [ 4026.473212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4026.482654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4028.009352] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4032.740467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4032.766488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4032.811119] Lustre: 250610:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4032.837387] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4032.839531] Lustre: Skipped 1 previous similar message [ 4032.876901] Lustre: lustre-MDT0001: new disk, initializing [ 4032.928444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4032.931931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4034.599730] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4038.092424] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4039.527568] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4042.133333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4042.162106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4042.274107] Lustre: lustre-OST0000: new disk, initializing [ 4042.276499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4042.314271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4042.317528] Lustre: Skipped 1 previous similar message [ 4043.077168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4043.081664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4043.128572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4044.503663] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4047.893669] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4048.615856] LustreError: 252976:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4053.109386] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4053.135660] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4053.192242] Lustre: lustre-OST0001: new disk, initializing [ 4053.194164] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4055.297030] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4058.498471] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4058.711122] LustreError: 252976:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4058.719546] LustreError: 252976:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4059.089123] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4062.216083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4062.220416] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4062.257461] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4068.807131] LustreError: 252976:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4072.417112] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4072.418784] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.420367] LustreError: Skipped 1 previous similar message [ 4072.425605] Lustre: Skipped 1 previous similar message [ 4072.426852] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4072.432176] Lustre: Skipped 3 previous similar messages [ 4075.628099] LustreError: 254107:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4075.631443] LustreError: 254107:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4075.654976] Lustre: server umount lustre-OST0001 complete [ 4077.538697] LustreError: 252345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4077.545573] LustreError: 252345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4082.658400] LustreError: 252344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4082.663499] LustreError: 252344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4087.779498] LustreError: 252346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4087.784907] LustreError: 252346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4089.312585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4089.312585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4089.312626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4089.315613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4089.317694] Lustre: Skipped 2 previous similar messages [ 4089.326894] Lustre: Skipped 2 previous similar messages [ 4092.899972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4092.902570] Lustre: Skipped 1 previous similar message [ 4093.997844] LustreError: 254332:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4094.000404] LustreError: 254332:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4094.042943] Lustre: server umount lustre-OST0000 complete [ 4098.016758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4098.017530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4098.020569] Lustre: Skipped 1 previous similar message [ 4101.663299] Lustre: server umount lustre-MDT0000 complete [ 4103.015225] LustreError: 250603:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762170482 with bad export cookie 3944763711649161234 [ 4103.018854] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4103.019292] LustreError: 250603:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4109.173649] Lustre: server umount lustre-MDT0001 complete [ 4113.255389] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 4114.428873] Key type lgssc unregistered [ 4114.573607] LNet: 255268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4114.576626] LNetError: 255268:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4114.583342] LNet: Removed LNI 192.168.202.137@tcp [ 4114.921034] Key type .llcrypt unregistered [ 4114.922570] Key type ._llcrypt unregistered [ 4126.663605] Key type ._llcrypt registered [ 4126.665345] Key type .llcrypt registered [ 4126.713438] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 4133.612231] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4134.101780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4134.112103] alg: No test for adler32 (adler32-zlib) [ 4134.974667] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 4135.068561] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 4136.655272] Key type lgssc registered [ 4137.099819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4140.554196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4143.123061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4145.402434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4147.701481] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4150.251407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4150.275440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4150.281292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4151.371120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4151.385363] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4151.423405] Lustre: lustre-MDT0000: new disk, initializing [ 4151.449306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4151.455758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4152.859733] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4157.313397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4157.337914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4157.379123] Lustre: 259192:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4157.406971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4157.409802] Lustre: Skipped 1 previous similar message [ 4157.449671] Lustre: lustre-MDT0001: new disk, initializing [ 4157.487836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4157.503068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4157.507250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4158.937000] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4162.161370] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4163.525901] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4165.890222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4165.912519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4166.007872] Lustre: lustre-OST0000: new disk, initializing [ 4166.009666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4166.036705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4166.773826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4166.777429] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4166.810876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4168.036303] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4171.330735] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4172.775222] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4172.847259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4174.285567] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4174.374365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4177.376856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4177.379557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4177.380704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4177.384615] Lustre: Skipped 1 previous similar message [ 4181.230083] LustreError: 262125:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4181.281056] Lustre: server umount lustre-OST0000 complete [ 4183.008209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4183.011509] LustreError: Skipped 1 previous similar message [ 4183.013433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4183.019654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4183.022062] Lustre: Skipped 1 previous similar message [ 4187.616615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4187.617426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4192.736300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4192.739298] Lustre: Skipped 2 previous similar messages [ 4197.343116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4197.378568] LustreError: 262326:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4197.380521] LustreError: 262326:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4197.411984] Lustre: server umount lustre-MDT0000 complete [ 4198.732765] LustreError: 259183:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762170578 with bad export cookie 13397918906427844399 [ 4198.734463] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4198.738196] LustreError: 259183:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4198.808880] LustreError: 262528:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4198.811506] LustreError: 262528:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4198.866808] Lustre: server umount lustre-MDT0001 complete [ 4201.796258] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 06:49:40 (1762170580) [ 4204.130353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4204.297426] LustreError: 263198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4204.304791] LustreError: 263198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4204.322448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4205.663783] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4208.521406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4208.626889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4209.915171] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4211.713943] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4212.908266] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4215.141439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4217.144450] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4218.991553] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4269.537740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4269.539311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4269.543034] Lustre: Skipped 1 previous similar message [ 4269.543346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4269.545722] Lustre: Skipped 1 previous similar message [ 4274.223087] LustreError: 265540:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4274.225369] LustreError: 265540:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4274.269978] Lustre: server umount lustre-OST0000 complete [ 4275.751520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4280.054274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4280.168947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4280.171897] Lustre: Skipped 1 previous similar message [ 4282.101558] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4283.940736] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4289.380450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4289.385720] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4289.385942] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4289.392923] Lustre: Skipped 1 previous similar message [ 4320.224823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4320.225438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4320.228056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4320.231095] Lustre: Skipped 1 previous similar message [ 4323.567243] LustreError: 267203:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4323.569863] LustreError: 267203:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4323.624369] Lustre: server umount lustre-OST0000 complete [ 4331.161215] Lustre: server umount lustre-MDT0000 complete [ 4332.580903] LustreError: 263177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762170712 with bad export cookie 13397918906427845267 [ 4332.585781] LustreError: 263177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4332.586039] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4332.655658] LustreError: 267608:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4332.658751] LustreError: 267608:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4332.726321] Lustre: server umount lustre-MDT0001 complete [ 4336.691925] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 4337.796346] Key type lgssc unregistered [ 4337.924548] LNet: 268139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4337.927616] LNetError: 268139:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4337.935568] LNet: Removed LNI 192.168.202.137@tcp [ 4338.245280] Key type .llcrypt unregistered [ 4338.246747] Key type ._llcrypt unregistered [ 4347.851402] Key type ._llcrypt registered [ 4347.852924] Key type .llcrypt registered [ 4347.894078] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4348.322641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4348.338295] alg: No test for adler32 (adler32-zlib) [ 4349.205414] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 4349.297405] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 4350.887171] Key type lgssc registered [ 4351.261323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4354.160601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4354.167806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4355.328817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4356.707444] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4359.745658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4359.864502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4361.223982] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4363.176142] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4364.471850] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4366.807632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4366.932072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4369.045417] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4371.002501] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4380.262766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4390.882759] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4390.883446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4390.884113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4390.885223] LustreError: Skipped 1 previous similar message [ 4390.893956] Lustre: Skipped 1 previous similar message [ 4395.502133] LustreError: 272552:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4395.546589] Lustre: server umount lustre-OST0000 complete [ 4401.121489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4401.121622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4401.129844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4401.132348] Lustre: Skipped 2 previous similar messages [ 4406.240276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4406.242299] Lustre: Skipped 1 previous similar message [ 4411.359258] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4411.360832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4411.364425] Lustre: Skipped 1 previous similar message [ 4411.391816] LustreError: 272753:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4411.393917] LustreError: 272753:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4411.425945] Lustre: server umount lustre-MDT0000 complete [ 4412.841347] LustreError: 269492:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762170792 with bad export cookie 66465153786813577 [ 4412.843491] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4412.845864] LustreError: 269492:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4412.917698] LustreError: 272956:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4412.920628] LustreError: 272956:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4412.982391] Lustre: server umount lustre-MDT0001 complete [ 4416.894343] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 4418.039606] Key type lgssc unregistered [ 4418.194521] LNet: 273487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4418.197378] LNetError: 273487:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4418.204397] LNet: Removed LNI 192.168.202.137@tcp [ 4418.543448] Key type .llcrypt unregistered [ 4418.545037] Key type ._llcrypt unregistered [ 4426.466251] Key type ._llcrypt registered [ 4426.467765] Key type .llcrypt registered [ 4426.517299] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4426.952747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4427.025419] alg: No test for adler32 (adler32-zlib) [ 4427.897093] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 4428.004581] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 4429.591189] Key type lgssc registered [ 4430.022198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4433.016205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4433.023284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4434.206461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4435.805906] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4439.226823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4439.392505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4440.956207] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4443.190709] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4444.699195] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4447.578763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4447.743929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4450.135783] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4452.347742] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4459.050914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4469.219865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4469.222965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4469.228144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4469.230856] Lustre: Skipped 1 previous similar message [ 4473.136067] LustreError: 277081:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4473.175818] Lustre: server umount lustre-OST0000 complete [ 4475.359889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4475.363064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4475.367624] Lustre: Skipped 1 previous similar message [ 4475.370222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4479.457552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4479.458102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4480.576448] LustreError: 277283:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4480.578658] LustreError: 277283:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4480.611334] Lustre: server umount lustre-MDT0000 complete [ 4482.117544] LustreError: 274649:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762170861 with bad export cookie 2205897417016689750 [ 4482.122065] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4482.122329] LustreError: 274649:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4482.196293] LustreError: 277485:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4482.199015] LustreError: 277485:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4482.266513] Lustre: server umount lustre-MDT0001 complete [ 4486.775865] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 4488.023669] Key type lgssc unregistered [ 4488.184555] LNet: 278016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4488.188310] LNetError: 278016:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4488.201467] LNet: Removed LNI 192.168.202.137@tcp [ 4488.553107] Key type .llcrypt unregistered [ 4488.554523] Key type ._llcrypt unregistered [ 4498.617190] Key type ._llcrypt registered [ 4498.618644] Key type .llcrypt registered [ 4498.665778] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4499.058594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4499.066987] alg: No test for adler32 (adler32-zlib) [ 4499.954977] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 4500.055322] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 4501.655199] Key type lgssc registered [ 4502.141197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4505.251178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4505.258923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.462641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4508.071457] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4511.548410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4511.705565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4513.252218] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4515.461800] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4517.012314] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4519.898192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4520.051537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4522.328157] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4524.412159] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4532.328380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4542.944993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4542.945101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4542.950313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4542.952910] LustreError: Skipped 1 previous similar message [ 4542.956080] Lustre: Skipped 1 previous similar message [ 4548.065078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4548.067294] Lustre: Skipped 1 previous similar message [ 4553.185268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4553.187275] Lustre: Skipped 1 previous similar message [ 4557.279113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4557.297363] LustreError: 282450:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4557.340110] Lustre: server umount lustre-OST0000 complete [ 4562.912082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4562.916218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4562.921589] Lustre: Skipped 1 previous similar message [ 4562.924275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4564.796355] LustreError: 282654:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4564.798775] LustreError: 282654:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4564.823274] Lustre: server umount lustre-MDT0000 complete [ 4566.172240] LustreError: 279393:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762170945 with bad export cookie 9378097266811655303 [ 4566.175409] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4566.177150] LustreError: 279393:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4566.243677] LustreError: 282856:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4566.245705] LustreError: 282856:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4566.299160] Lustre: server umount lustre-MDT0001 complete [ 4570.135764] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 4571.215621] Key type lgssc unregistered [ 4571.352452] LNet: 283386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4571.355341] LNetError: 283386:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4571.362503] LNet: Removed LNI 192.168.202.137@tcp [ 4571.673590] Key type .llcrypt unregistered [ 4571.674927] Key type ._llcrypt unregistered [ 4579.181063] Key type ._llcrypt registered [ 4579.182658] Key type .llcrypt registered [ 4579.237974] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4579.579031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4579.585899] alg: No test for adler32 (adler32-zlib) [ 4580.440118] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 4580.531642] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 4582.119176] Key type lgssc registered [ 4582.550505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4585.613158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4585.623824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4586.831728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4588.274685] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4591.567605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4591.708439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4593.146624] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4595.199149] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4596.570779] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4599.234277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4599.366287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4601.569692] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4603.655213] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4611.686359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4622.307749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4622.309245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4622.309905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4622.316833] Lustre: Skipped 1 previous similar message [ 4623.662106] LustreError: 287028:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4623.704422] Lustre: server umount lustre-OST0000 complete [ 4627.424597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4627.425106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4627.425850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4627.425854] Lustre: Skipped 1 previous similar message [ 4627.428261] LustreError: Skipped 1 previous similar message [ 4627.432780] Lustre: Skipped 1 previous similar message [ 4631.170305] LustreError: 287231:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4631.173212] LustreError: 287231:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4631.204442] Lustre: server umount lustre-MDT0000 complete [ 4632.544438] LustreError: 284605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4632.551784] LustreError: 284605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4632.661741] LustreError: 284561:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762171012 with bad export cookie 4710839748485739137 [ 4632.666617] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4632.667907] LustreError: 284561:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4632.732480] LustreError: 287432:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4632.734438] LustreError: 287432:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4632.797032] Lustre: server umount lustre-MDT0001 complete [ 4636.784856] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 4637.954304] Key type lgssc unregistered [ 4638.104626] LNet: 287964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4638.108725] LNetError: 287964:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4639.141412] LNet: Removed LNI 192.168.202.137@tcp [ 4639.449384] Key type .llcrypt unregistered [ 4639.451109] Key type ._llcrypt unregistered [ 4643.201393] Key type ._llcrypt registered [ 4643.202674] Key type .llcrypt registered [ 4643.245187] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4649.154198] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 4655.012891] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4655.327179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4655.334720] alg: No test for adler32 (adler32-zlib) [ 4656.195060] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 4656.277361] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 4657.863198] Key type lgssc registered [ 4658.262967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4661.447479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4663.638048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4665.804708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4667.976447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4670.419051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4670.439145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4670.443979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4671.534768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4671.548206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4671.590399] Lustre: lustre-MDT0000: new disk, initializing [ 4671.634186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4671.644506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4673.059843] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4677.316425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4677.336044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4677.366133] Lustre: 291446:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4677.389555] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4677.391775] Lustre: Skipped 1 previous similar message [ 4677.426142] Lustre: lustre-MDT0001: new disk, initializing [ 4677.458928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4677.472801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4677.477423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4678.760926] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4681.838886] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4683.016452] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4685.414455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4685.445261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4685.552899] Lustre: lustre-OST0000: new disk, initializing [ 4685.555362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4685.587168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4686.022529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4686.027148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4686.065980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4687.697209] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4690.966515] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4692.431480] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4692.512222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4693.889087] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4693.985121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4696.545030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4696.545169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4696.547267] LustreError: Skipped 1 previous similar message [ 4696.548506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4696.550857] Lustre: Skipped 1 previous similar message [ 4696.750091] LustreError: 294383:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4696.792695] Lustre: server umount lustre-OST0000 complete [ 4698.079818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4698.082873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4698.088084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4698.089963] Lustre: Skipped 1 previous similar message [ 4701.665114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4701.665886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4704.189471] LustreError: 294584:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4704.191626] LustreError: 294584:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4704.222436] Lustre: server umount lustre-MDT0000 complete [ 4705.564622] LustreError: 291437:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762171085 with bad export cookie 14908765091992820853 [ 4705.567446] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4705.568817] LustreError: 291437:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4705.636720] LustreError: 294785:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4705.639711] LustreError: 294785:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4705.699583] Lustre: server umount lustre-MDT0001 complete [ 4708.752193] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 06:58:07 (1762171087) [ 4711.153926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4711.328197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4712.715526] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4715.766545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4715.890337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4717.256621] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4719.039378] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4720.335847] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4722.738348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4724.690619] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4726.530484] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4741.603840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4741.604545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4741.608614] LustreError: Skipped 1 previous similar message [ 4741.614358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4741.617579] Lustre: Skipped 2 previous similar messages [ 4745.774152] LustreError: 297729:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4745.776398] LustreError: 297729:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4745.818494] Lustre: server umount lustre-OST0000 complete [ 4749.793965] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4749.797381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4749.801522] Lustre: Skipped 1 previous similar message [ 4749.803824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4753.471196] LustreError: 297931:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4753.473782] LustreError: 297931:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4753.504779] Lustre: server umount lustre-MDT0000 complete [ 4754.830145] LustreError: 295436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762171134 with bad export cookie 14908765091992821721 [ 4754.833078] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4754.834252] LustreError: 295436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4754.838795] LustreError: 296177:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4754.960668] Lustre: server umount lustre-MDT0001 complete [ 4758.530361] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 4759.559123] Key type lgssc unregistered [ 4759.691516] LNet: 298665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4759.695557] LNetError: 298665:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4759.705592] LNet: Removed LNI 192.168.202.137@tcp [ 4760.028775] Key type .llcrypt unregistered [ 4760.030288] Key type ._llcrypt unregistered [ 4768.786967] Key type ._llcrypt registered [ 4768.788662] Key type .llcrypt registered [ 4768.831569] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 4774.944510] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4775.246666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4775.287031] alg: No test for adler32 (adler32-zlib) [ 4776.164252] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 4776.257503] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 4777.847171] Key type lgssc registered [ 4778.221618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4781.124860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4783.276047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4785.246304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4787.385712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4789.080180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4790.820819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4793.142901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4793.165157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4793.169573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4794.247147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4794.259559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4794.299263] Lustre: lustre-MDT0000: new disk, initializing [ 4794.328028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4794.336092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4795.655359] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4799.805043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4799.826587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4799.862448] Lustre: 302441:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4799.886866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4799.888546] Lustre: Skipped 1 previous similar message [ 4799.923466] Lustre: lustre-MDT0001: new disk, initializing [ 4799.955397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4799.968650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4799.972871] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4801.358232] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4804.538628] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4805.677300] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4807.814350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4807.840596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4807.948151] Lustre: lustre-OST2710: new disk, initializing [ 4807.950921] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4807.986694] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4809.868155] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4810.677420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4810.680215] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4810.708641] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4812.981552] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4815.204874] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4815.229530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4815.278907] Lustre: lustre-OST03e8: new disk, initializing [ 4815.281198] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4815.308123] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4816.840603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4816.846116] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4816.881292] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4817.157907] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4818.890305] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4821.403360] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4821.479780] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4822.795453] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4822.881457] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4824.134988] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4824.203188] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4825.459991] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4825.547315] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4827.105210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4827.107087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4827.108901] Lustre: Skipped 1 previous similar message [ 4832.225110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4832.227503] Lustre: Skipped 3 previous similar messages [ 4832.637643] LustreError: 306787:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4832.671417] Lustre: server umount lustre-MDT0000 complete [ 4833.910256] LustreError: 304188:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762171213 with bad export cookie 17083812933508511166 [ 4833.913949] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4833.914144] LustreError: 304188:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4833.981166] LustreError: 306989:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4833.983968] LustreError: 306989:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4834.042794] Lustre: server umount lustre-MDT0001 complete [ 4845.555082] LustreError: 307189:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4845.557545] LustreError: 307189:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4845.577955] Lustre: server umount lustre-OST2710 complete [ 4857.137642] LustreError: 307391:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4857.139691] LustreError: 307391:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4857.182133] Lustre: server umount lustre-OST03e8 complete [ 4863.607863] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 4866.903842] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4870.473416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4872.753845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4875.025481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4877.147154] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4879.347489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4879.364950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4879.437340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4879.446878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4879.479235] Lustre: lustre-MDT0000: new disk, initializing [ 4879.498732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4879.504375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4880.782457] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4884.859626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4884.882813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4884.914778] Lustre: 310183:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4885.017532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4886.291303] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4889.491433] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4890.727913] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4892.931515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4892.951858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4893.030856] Lustre: lustre-OST0000: new disk, initializing [ 4893.032131] Lustre: Skipped 1 previous similar message [ 4893.033614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4893.035560] Lustre: Skipped 2 previous similar messages [ 4893.059799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4893.062392] Lustre: Skipped 1 previous similar message [ 4894.876615] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4896.758318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4896.760886] Lustre: Skipped 1 previous similar message [ 4896.763039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4896.800403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4897.937934] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4899.222943] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4899.299217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4900.507180] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4900.581194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4901.857219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4901.857857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4901.859765] LustreError: Skipped 1 previous similar message [ 4901.863470] Lustre: Skipped 2 previous similar messages [ 4901.864046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4901.867202] Lustre: Skipped 1 previous similar message [ 4903.215095] LustreError: 313129:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4903.217100] LustreError: 313129:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4903.257938] Lustre: server umount lustre-OST0000 complete [ 4905.439938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4905.442091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4905.445706] Lustre: Skipped 1 previous similar message [ 4905.447558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4910.750922] Lustre: server umount lustre-MDT0000 complete [ 4912.112246] LustreError: 310973:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762171291 with bad export cookie 17083812933508512832 [ 4912.115218] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4912.117260] LustreError: 310973:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4912.196742] LustreError: 313534:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4912.199955] LustreError: 313534:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4912.259741] Lustre: server umount lustre-MDT0001 complete [ 4915.422662] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 07:01:34 (1762171294) [ 4915.955842] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4916.546985] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 07:01:35 (1762171295) [ 4917.612957] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4920.495433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4921.780504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4922.991685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4924.221703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4924.610568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4926.923203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4927.021186] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4927.031638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4927.094543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4928.484741] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4931.530897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4931.566394] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4931.577256] Lustre: 315660:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4933.086315] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4934.946555] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4936.155242] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4938.476462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4938.504957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4938.576923] Lustre: Denying initial registration attempt from nid 192.168.202.137@tcp, specified as failover [ 4938.582034] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4938.587475] LustreError: 317276:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 4938.590111] LustreError: 317276:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4938.592371] LustreError: 317276:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4938.617573] Lustre: server umount lustre-OST0000 complete [ 4938.618798] LustreError: 317276:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4941.791977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4941.795654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4941.801449] Lustre: Skipped 1 previous similar message [ 4941.804074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4941.806882] Lustre: Skipped 2 previous similar messages [ 4947.071120] LustreError: 317643:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4947.073492] LustreError: 317643:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4947.099573] Lustre: server umount lustre-MDT0000 complete [ 4948.489216] LustreError: 315650:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762171328 with bad export cookie 17083812933508513700 [ 4948.492651] LustreError: 315650:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4948.492885] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4948.609847] Lustre: server umount lustre-MDT0001 complete [ 4952.319341] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 4953.434230] Key type lgssc unregistered [ 4953.558428] LNet: 318376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4953.561377] LNetError: 318376:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4953.569390] LNet: Removed LNI 192.168.202.137@tcp [ 4953.889728] Key type .llcrypt unregistered [ 4953.890812] Key type ._llcrypt unregistered [ 4957.796111] Key type ._llcrypt registered [ 4957.797238] Key type .llcrypt registered [ 4957.835043] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4958.184366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4958.212973] alg: No test for adler32 (adler32-zlib) [ 4959.079572] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 4959.168677] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 4960.751160] Key type lgssc registered [ 4961.153415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4963.612850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4965.039806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4966.369459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4967.766840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4968.139247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4974.791372] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4978.268291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4978.273888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4979.380471] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4979.393920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4979.468596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4980.866780] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4983.719964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4983.751145] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4983.761451] Lustre: 321106:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4983.852422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4985.177861] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4986.941291] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4988.115445] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4990.321663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4990.346179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4990.404219] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4990.461103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4992.298222] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4994.056744] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4999.649051] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4999.649932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4999.650711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4999.650873] LustreError: Skipped 1 previous similar message [ 4999.656848] Lustre: Skipped 1 previous similar message [ 5000.813455] LustreError: 323392:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5000.848125] Lustre: server umount lustre-OST0000 complete [ 5004.255694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5004.257722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5004.261159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5004.262628] Lustre: Skipped 1 previous similar message [ 5008.123040] LustreError: 323594:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5008.124715] LustreError: 323594:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5008.150209] Lustre: server umount lustre-MDT0000 complete [ 5009.334323] LustreError: 321098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762171388 with bad export cookie 15747959930334478398 [ 5009.337383] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5009.338420] LustreError: 321098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5009.393423] LustreError: 323795:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5009.395142] LustreError: 323795:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5009.443694] Lustre: server umount lustre-MDT0001 complete [ 5012.927069] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 5013.915730] Key type lgssc unregistered [ 5014.042520] LNet: 324326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5014.045023] LNetError: 324326:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5014.053375] LNet: Removed LNI 192.168.202.137@tcp [ 5014.335384] Key type .llcrypt unregistered [ 5014.336573] Key type ._llcrypt unregistered [ 5022.638355] Key type ._llcrypt registered [ 5022.639627] Key type .llcrypt registered [ 5022.670692] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5022.946701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5022.980044] alg: No test for adler32 (adler32-zlib) [ 5023.821266] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 5023.895813] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 5025.471218] Key type lgssc registered [ 5025.846857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5028.302210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5028.307762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5029.465959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5030.664926] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5033.406760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5033.508803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5034.679177] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5036.368411] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5037.656676] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5039.877785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5039.984574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5041.849397] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5043.632885] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5064.160032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5064.162722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5064.167240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5065.696659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5065.697887] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5065.699709] Lustre: Skipped 2 previous similar messages [ 5065.703663] Lustre: Skipped 1 previous similar message [ 5067.842306] LustreError: 327969:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5067.876962] Lustre: server umount lustre-MDT0000 complete [ 5069.323034] LustreError: 325680:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762171448 with bad export cookie 3488015799140019591 [ 5069.327242] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5069.328476] LustreError: 325680:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5069.697696] LustreError: 325694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5069.705389] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.37@tcp (stopping) [ 5069.705905] LustreError: 325694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5069.707957] Lustre: Skipped 1 previous similar message [ 5070.816113] LustreError: 327852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5070.816309] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5074.817680] LustreError: 327852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5074.818346] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.37@tcp (stopping) [ 5074.825232] Lustre: Skipped 1 previous similar message [ 5075.452989] LustreError: 328176:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5075.455045] LustreError: 328176:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5075.501697] Lustre: server umount lustre-MDT0001 complete [ 5077.745231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5080.788152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5094.367628] LustreError: 325009:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007e15b655 x1847770641097856/t0(0) o250->MGC192.168.202.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5094.483437] LustreError: 327309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5094.489785] LustreError: 327309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5094.511316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5094.531979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 5095.850300] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5098.592630] LustreError: 329012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5098.598544] LustreError: 329012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5098.810498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5098.932573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5100.207841] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5101.952642] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5104.041273] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5106.668202] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5107.892854] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5108.461746] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5110.240108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5110.242507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5110.246990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5110.249836] Lustre: Skipped 1 previous similar message [ 5114.339621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5114.341780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5119.457543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5119.460152] Lustre: Skipped 3 previous similar messages [ 5124.063121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5124.086798] LustreError: 330478:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5124.090220] LustreError: 330478:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5124.130609] Lustre: server umount lustre-OST0000 complete [ 5129.696793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5129.697126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5129.700702] Lustre: Skipped 1 previous similar message [ 5131.523268] LustreError: 330681:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5131.525037] LustreError: 330681:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5131.553934] Lustre: server umount lustre-MDT0000 complete [ 5132.817706] LustreError: 325681:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762171512 with bad export cookie 3488015799140026745 [ 5132.821285] LustreError: 325681:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5132.822307] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5132.943720] Lustre: server umount lustre-MDT0001 complete [ 5136.367050] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 5137.377764] Key type lgssc unregistered [ 5137.509465] LNet: 331413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5137.511911] LNetError: 331413:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5137.520434] LNet: Removed LNI 192.168.202.137@tcp [ 5137.839136] Key type .llcrypt unregistered [ 5137.840497] Key type ._llcrypt unregistered [ 5147.551308] Key type ._llcrypt registered [ 5147.553138] Key type .llcrypt registered [ 5147.599339] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5148.050409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5148.065124] alg: No test for adler32 (adler32-zlib) [ 5148.921517] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 5149.014610] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 5150.599140] Key type lgssc registered [ 5151.036031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5153.639650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5153.644787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5154.782241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5155.959788] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5158.612573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5158.734174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5159.995140] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5161.701361] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5162.807964] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5163.779465] LustreError: 332815:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5163.781893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5163.782586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5163.782664] LustreError: 332815:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5163.785717] Lustre: Skipped 1 previous similar message [ 5163.794981] LustreError: 332815:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 5168.928358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5168.930774] Lustre: Skipped 2 previous similar messages [ 5174.239975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5174.241700] Lustre: Skipped 1 previous similar message [ 5178.335127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5178.369180] LustreError: 334174:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5178.396457] Lustre: server umount lustre-MDT0000 complete [ 5179.360288] LustreError: 332787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5179.365808] LustreError: 332787:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5179.677408] LustreError: 334175:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762171559 with bad export cookie 17834586012519037438 [ 5179.680499] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5179.681583] LustreError: 334175:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5179.740175] LustreError: 334379:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5179.742685] LustreError: 334379:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5179.797381] Lustre: server umount lustre-MDT0001 complete [ 5184.636481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5184.727350] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5184.737962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5184.809592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5186.100187] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5188.944554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5188.975528] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5188.988340] Lustre: 335148:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5189.101994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5190.428127] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5192.287104] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5193.554464] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5196.310673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5196.385478] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5196.389363] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5196.393034] LustreError: 336805:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 5196.396243] LustreError: 336805:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 5196.398067] LustreError: 336805:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 5196.422683] Lustre: server umount lustre-OST0000 complete [ 5196.424087] LustreError: 336805:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5199.382444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5199.454985] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5199.528907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5201.536515] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5203.454316] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5206.177568] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5206.201873] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5206.225204] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5206.253213] Lustre: lustre-OST0001: new disk, initializing [ 5206.255416] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5208.197936] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5208.971407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5208.977152] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5208.995395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 5209.021583] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5210.011564] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5214.179317] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5214.179957] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5214.182918] LustreError: Skipped 1 previous similar message [ 5214.186977] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5214.190610] Lustre: Skipped 1 previous similar message [ 5217.134100] LustreError: 338922:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5217.137639] LustreError: 338922:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5217.157636] Lustre: server umount lustre-OST0001 complete [ 5219.297561] LustreError: 338691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5219.298091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5219.298118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5219.300870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5219.303429] LustreError: 338691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5219.307537] Lustre: Skipped 2 previous similar messages [ 5224.419219] LustreError: 337271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5224.427785] LustreError: 337271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5224.558120] LustreError: 339123:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5224.560603] LustreError: 339123:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5224.604293] Lustre: server umount lustre-OST0000 complete [ 5229.536948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.537666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5229.540381] Lustre: Skipped 1 previous similar message [ 5229.543716] Lustre: Skipped 4 previous similar messages [ 5240.287087] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5240.324837] LustreError: 339324:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5240.326528] LustreError: 339324:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5240.355724] Lustre: server umount lustre-MDT0000 complete [ 5241.674096] LustreError: 335184:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762171621 with bad export cookie 17834586012519037956 [ 5241.677921] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5241.679352] LustreError: 335184:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5241.811645] Lustre: server umount lustre-MDT0001 complete [ 5245.501406] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 5246.591935] Key type lgssc unregistered [ 5246.727367] LNet: 340060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5246.729614] LNetError: 340060:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5246.742307] LNet: Removed LNI 192.168.202.137@tcp [ 5247.030153] Key type .llcrypt unregistered [ 5247.031325] Key type ._llcrypt unregistered [ 5258.515422] Key type ._llcrypt registered [ 5258.517208] Key type .llcrypt registered [ 5258.557054] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 5264.926822] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5265.337437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5265.361111] alg: No test for adler32 (adler32-zlib) [ 5266.222659] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 5266.308376] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 5267.895148] Key type lgssc registered [ 5268.269578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5271.296416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5273.613675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5275.733247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5277.944197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5280.199965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5280.221825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5280.226871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5281.312598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5281.324677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5281.362287] Lustre: lustre-MDT0000: new disk, initializing [ 5281.386664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5281.393457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5282.789926] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5286.998913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5287.018163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5287.052959] Lustre: 343980:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5287.078659] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5287.081202] Lustre: Skipped 1 previous similar message [ 5287.122626] Lustre: lustre-MDT0001: new disk, initializing [ 5287.155661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5287.168601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5287.172519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5288.468681] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5291.447892] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5292.541350] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5294.680325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5294.698896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5294.780775] Lustre: lustre-OST0000: new disk, initializing [ 5294.783233] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5294.808936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5296.605097] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5296.699879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5296.702960] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5296.751250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5299.603277] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5300.909360] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5300.979544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5302.268440] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5302.339142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5305.004069] LustreError: 346915:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5305.038601] Lustre: server umount lustre-OST0000 complete [ 5307.360399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5307.361090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5307.364273] Lustre: Skipped 1 previous similar message [ 5307.366981] Lustre: Skipped 1 previous similar message [ 5312.480101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5312.482442] Lustre: Skipped 1 previous similar message [ 5315.555610] LustreError: 347117:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5315.557962] LustreError: 347117:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5315.585129] Lustre: server umount lustre-MDT0000 complete [ 5316.796430] LustreError: 343974:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762171696 with bad export cookie 14937428529472657448 [ 5316.799235] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5316.800104] LustreError: 343974:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5327.332651] LustreError: 347319:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 5327.335793] LustreError: 347319:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5327.387623] Lustre: server umount lustre-MDT0001 complete [ 5330.583916] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 07:08:29 (1762171709) [ 5332.691470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5334.669840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5342.448738] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 5345.597478] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5348.921065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5351.060769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5353.123576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5355.239542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5357.595376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5357.618556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5357.715030] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5357.728292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5357.770258] Lustre: lustre-MDT0000: new disk, initializing [ 5357.798325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5357.804834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5359.185835] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5363.567676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5363.588267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5363.625037] Lustre: 351093:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5363.650613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5363.652691] Lustre: Skipped 1 previous similar message [ 5363.691548] Lustre: lustre-MDT0001: new disk, initializing [ 5363.727529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5363.741369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5363.745029] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5365.113759] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5368.265183] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5369.530052] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5371.937409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5371.958768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5372.057118] Lustre: lustre-OST0000: new disk, initializing [ 5372.059109] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5372.086118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5372.799636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5372.802345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5372.848425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5374.151344] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5377.342581] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5378.789689] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5378.870154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5380.237507] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5380.311544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5383.138087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5383.138919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5383.144101] Lustre: Skipped 1 previous similar message [ 5383.147688] LustreError: Skipped 1 previous similar message [ 5383.147971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5383.153368] Lustre: Skipped 1 previous similar message [ 5387.053093] LustreError: 354029:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5387.055563] LustreError: 354029:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5387.095369] Lustre: server umount lustre-OST0000 complete [ 5389.279830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5389.282782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5389.287152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5393.376259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5393.376619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5394.562661] LustreError: 354230:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5394.564373] LustreError: 354230:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5394.601823] Lustre: server umount lustre-MDT0000 complete [ 5395.905328] LustreError: 351871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762171775 with bad export cookie 14937428529472658260 [ 5395.908647] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5395.910956] LustreError: 351871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5396.050540] Lustre: server umount lustre-MDT0001 complete [ 5399.003480] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 07:09:37 (1762171777) [ 5401.561336] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 07:09:40 (1762171780) [ 5404.458565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5404.657114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.048474] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5409.064673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5410.599257] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5412.455740] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5413.684143] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5416.074665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5418.178662] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5420.135703] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5434.073820] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5434.708888] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5435.263467] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5435.866456] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5436.545851] Lustre: Failing over lustre-MDT0000 [ 5436.683944] LustreError: 357907:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5436.686050] LustreError: 357907:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5436.719714] Lustre: server umount lustre-MDT0000 complete [ 5439.968255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5439.971583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5439.976749] LustreError: 356159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5439.982030] LustreError: 356159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5440.909193] LustreError: 355442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5446.018322] LustreError: 355444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5446.024663] LustreError: 355444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5449.682035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5449.720508] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5449.809394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5449.812375] Lustre: Skipped 2 previous similar messages [ 5449.836179] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5451.046391] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5451.137235] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5454.827675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5454.832369] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5454.850899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5456.027909] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5456.618973] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5458.508886] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5459.195061] Lustre: Failing over lustre-MDT0000 [ 5459.366335] LustreError: 358987:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5459.367961] LustreError: 358987:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5459.399742] Lustre: server umount lustre-MDT0000 complete [ 5459.936665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5459.937432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5459.938585] LustreError: 355442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5459.938591] LustreError: 355442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5459.948863] Lustre: Skipped 3 previous similar messages [ 5465.056204] LustreError: 355447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5465.060792] LustreError: 355447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5472.407881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5472.449644] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5472.562614] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5473.971916] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5476.738270] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5477.860809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5477.862593] Lustre: Skipped 2 previous similar messages [ 5477.868814] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5477.885706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5479.069062] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5479.650397] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5482.978149] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5482.978806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5482.979617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5482.979623] Lustre: Skipped 1 previous similar message [ 5482.981767] Lustre: Skipped 3 previous similar messages [ 5488.091469] Lustre: server umount lustre-MDT0000 complete [ 5488.097630] LustreError: 356159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5488.103462] LustreError: 356159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5489.336314] LustreError: 355427:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762171868 with bad export cookie 14937428529472660871 [ 5489.338074] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5489.341474] LustreError: 355427:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5489.457788] Lustre: server umount lustre-MDT0001 complete [ 5501.045154] LustreError: 360416:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5501.048280] LustreError: 360416:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5501.098461] Lustre: server umount lustre-OST0000 complete [ 5506.362882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5506.508412] LustreError: 361244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5507.819157] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5510.806845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5512.292687] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5514.144745] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5515.417410] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5517.720602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5517.826468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5517.828645] Lustre: Skipped 3 previous similar messages [ 5519.702472] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5521.572163] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5532.198987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5541.343896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5541.346751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5541.352237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5541.353930] Lustre: Skipped 2 previous similar messages [ 5544.531145] Lustre: server umount lustre-OST0000 complete [ 5560.287151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5560.346512] Lustre: server umount lustre-MDT0000 complete [ 5561.671104] LustreError: 361223:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762171941 with bad export cookie 14937428529472661578 [ 5561.675084] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5561.675212] LustreError: 361223:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5561.805374] Lustre: server umount lustre-MDT0001 complete [ 5565.566680] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 5566.596537] Key type lgssc unregistered [ 5566.730354] LNet: 364446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5566.733287] LNetError: 364446:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5566.740452] LNet: Removed LNI 192.168.202.137@tcp [ 5567.070829] Key type .llcrypt unregistered [ 5567.071961] Key type ._llcrypt unregistered [ 5577.142701] Key type ._llcrypt registered [ 5577.144211] Key type .llcrypt registered [ 5577.187288] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5577.579863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5577.653231] alg: No test for adler32 (adler32-zlib) [ 5578.517276] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 5578.609820] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 5580.199139] Key type lgssc registered [ 5580.599637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5583.444393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5583.451443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5584.621920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5586.069331] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5588.921308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5589.051498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5590.335647] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5592.132873] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5593.297433] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5595.503963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5595.612804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5597.455143] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5599.149735] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5609.957376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5619.679905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5619.683585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5619.689698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5620.192219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5620.192980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5620.194554] Lustre: Skipped 1 previous similar message [ 5620.198133] Lustre: Skipped 1 previous similar message [ 5624.829168] LustreError: 368187:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5624.854723] Lustre: server umount lustre-MDT0000 complete [ 5625.312183] LustreError: 365939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5625.318444] LustreError: 365939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5625.961054] LustreError: 367539:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172005 with bad export cookie 14396422207020784561 [ 5625.963754] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5625.965555] LustreError: 367539:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5626.034769] LustreError: 368390:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5626.037261] LustreError: 368390:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5626.080878] Lustre: server umount lustre-MDT0001 complete [ 5636.404206] LustreError: 368591:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5636.406148] LustreError: 368591:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5636.451262] Lustre: server umount lustre-OST0000 complete [ 5640.998950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5641.156289] LustreError: 369320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5641.181288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5642.474253] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5645.359738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5645.499367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5646.863589] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5648.726338] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5649.993970] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5652.251770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5654.182715] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5655.934079] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5665.702456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5676.002633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5676.003563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5676.006745] Lustre: Skipped 2 previous similar messages [ 5676.008242] Lustre: Skipped 2 previous similar messages [ 5679.679360] LustreError: 371690:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5679.681053] LustreError: 371690:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5679.711348] Lustre: server umount lustre-MDT0000 complete [ 5681.048951] LustreError: 369300:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172060 with bad export cookie 14396422207020785653 [ 5681.053478] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5681.053641] LustreError: 369300:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5681.120162] LustreError: 369315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5681.120527] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5681.128229] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5681.129914] Lustre: Skipped 1 previous similar message [ 5681.196455] Lustre: server umount lustre-MDT0001 complete [ 5688.556280] LustreError: 372093:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5688.558833] LustreError: 372093:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5688.603348] Lustre: server umount lustre-OST0000 complete [ 5693.864630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5694.020354] LustreError: 372922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5694.046550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5694.048487] Lustre: Skipped 1 previous similar message [ 5695.416088] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5698.367391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5699.820638] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5701.695766] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5702.899907] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5705.149106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5707.188988] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5709.067838] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5718.568115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5723.619129] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5723.621715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5723.623287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5723.626210] Lustre: Skipped 1 previous similar message [ 5723.629226] Lustre: Skipped 1 previous similar message [ 5726.831184] LustreError: 375190:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5726.836161] LustreError: 375190:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5726.882454] Lustre: server umount lustre-OST0000 complete [ 5733.856447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5733.858532] Lustre: Skipped 3 previous similar messages [ 5734.354108] Lustre: server umount lustre-MDT0000 complete [ 5735.628339] LustreError: 375091:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172115 with bad export cookie 14396422207020786668 [ 5735.630928] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5735.631767] LustreError: 375091:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5735.749919] Lustre: server umount lustre-MDT0001 complete [ 5739.476868] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 5740.554126] Key type lgssc unregistered [ 5740.661264] LNet: 376125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5740.663478] LNetError: 376125:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5740.670274] LNet: Removed LNI 192.168.202.137@tcp [ 5740.945712] Key type .llcrypt unregistered [ 5740.946803] Key type ._llcrypt unregistered [ 5753.485345] Key type ._llcrypt registered [ 5753.490295] Key type .llcrypt registered [ 5753.560416] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5754.438346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5754.459540] alg: No test for adler32 (adler32-zlib) [ 5755.443921] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 5755.625434] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 5757.303167] Key type lgssc registered [ 5758.063611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5763.448420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5763.478230] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5763.482586] LustreError: 377523:0:(osd_handler.c:8570:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5763.501226] LustreError: 377523:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5763.508131] LustreError: 377523:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5763.518969] LustreError: 377523:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5763.529832] LustreError: 377523:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5768.713069] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5768.716579] LustreError: 377920:0:(osd_handler.c:8570:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5768.737742] LustreError: 377920:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5768.744510] LustreError: 377920:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5768.749360] LustreError: 377920:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5768.756282] LustreError: 377920:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5778.352625] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 5779.495131] Key type lgssc unregistered [ 5779.615344] LNet: 378905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5779.618254] LNetError: 378905:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5779.625382] LNet: Removed LNI 192.168.202.137@tcp [ 5779.921578] Key type .llcrypt unregistered [ 5779.923284] Key type ._llcrypt unregistered [ 5786.018674] Key type ._llcrypt registered [ 5786.019680] Key type .llcrypt registered [ 5786.056188] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 5791.996589] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5792.295894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5792.302640] alg: No test for adler32 (adler32-zlib) [ 5793.143268] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 5793.214081] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 5794.791123] Key type lgssc registered [ 5795.139135] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5798.259280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5800.244072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5802.170771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5804.250962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5806.504495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5806.525565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5806.530077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5807.620335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5807.631749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5807.671264] Lustre: lustre-MDT0000: new disk, initializing [ 5807.700151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5807.708206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5809.032039] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5813.286640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5813.308469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5813.340945] Lustre: 382010:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5813.365780] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5813.368922] Lustre: Skipped 1 previous similar message [ 5813.404157] Lustre: lustre-MDT0001: new disk, initializing [ 5813.434085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5813.447293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5813.450749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5814.781536] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5818.012367] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5819.190087] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5821.560129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5821.586344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5821.683709] Lustre: lustre-OST0000: new disk, initializing [ 5821.686367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5821.717693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5823.657262] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5824.693955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5824.698340] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5824.737863] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5826.681338] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5827.974090] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5828.048623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5829.250390] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5829.319753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5830.114334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5830.114756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5830.116444] LustreError: Skipped 1 previous similar message [ 5830.122985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5830.125423] Lustre: Skipped 1 previous similar message [ 5835.235398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5835.237787] Lustre: Skipped 1 previous similar message [ 5836.014081] LustreError: 384945:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5836.043990] Lustre: server umount lustre-OST0000 complete [ 5839.327793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5839.331200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5839.335928] Lustre: Skipped 1 previous similar message [ 5839.338028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5840.352807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5843.388061] LustreError: 385147:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5843.390552] LustreError: 385147:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5843.412723] Lustre: server umount lustre-MDT0000 complete [ 5844.569054] LustreError: 382000:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172224 with bad export cookie 328806700985785457 [ 5844.571921] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5844.572158] LustreError: 382000:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5844.634459] LustreError: 385348:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5844.636204] LustreError: 385348:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5844.684084] Lustre: server umount lustre-MDT0001 complete [ 5847.310732] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5847.821823] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 07:17:06 (1762172226) [ 5850.024755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5850.234313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5851.552976] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5854.522308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5854.624795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5855.852139] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5857.548605] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5858.604879] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5860.749493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5862.653224] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5864.368090] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5866.571426] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5866.593059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5866.644442] Lustre: lustre-OST0001: new disk, initializing [ 5866.647442] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5866.677927] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5866.680374] Lustre: Skipped 1 previous similar message [ 5868.570105] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5871.687822] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5872.127305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5872.130810] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5872.156130] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5874.596352] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5874.598238] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5874.602478] Lustre: Skipped 1 previous similar message [ 5874.605209] Lustre: Skipped 2 previous similar messages [ 5879.671137] LustreError: 389264:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5879.674940] LustreError: 389264:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5879.711406] Lustre: server umount lustre-OST0001 complete [ 5879.783785] LustreError: 387661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5879.799416] LustreError: 387661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5884.898502] LustreError: 388276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5884.920089] LustreError: 388276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5890.021852] LustreError: 387664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5890.033497] LustreError: 387664:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5895.138810] LustreError: 387664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5895.140167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5895.148233] LustreError: 387664:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5895.158113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5895.160188] Lustre: Skipped 1 previous similar message [ 5900.084118] LustreError: 389491:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5900.091448] LustreError: 389491:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5900.225581] Lustre: server umount lustre-OST0000 complete [ 5905.377160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5905.380785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5905.382444] Lustre: Skipped 2 previous similar messages [ 5905.388229] Lustre: Skipped 1 previous similar message [ 5917.151103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5917.188460] LustreError: 389692:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5917.191192] LustreError: 389692:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5917.224604] Lustre: server umount lustre-MDT0000 complete [ 5918.627746] LustreError: 388567:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172298 with bad export cookie 328806700985786325 [ 5918.630743] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5918.634771] LustreError: 388567:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5924.807623] Lustre: server umount lustre-MDT0001 complete [ 5928.647386] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 5929.760721] Key type lgssc unregistered [ 5929.899329] LNet: 390426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5929.902653] LNetError: 390426:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5929.911331] LNet: Removed LNI 192.168.202.137@tcp [ 5930.239635] Key type .llcrypt unregistered [ 5930.241207] Key type ._llcrypt unregistered [ 5943.213506] Key type ._llcrypt registered [ 5943.214997] Key type .llcrypt registered [ 5943.259105] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 5950.258825] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5950.732148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5950.752262] alg: No test for adler32 (adler32-zlib) [ 5951.654063] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 5951.769272] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 5953.359128] Key type lgssc registered [ 5953.810181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5957.748511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5960.516355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5963.017308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5965.606947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5968.363088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5968.387777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5968.394071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5969.498086] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5969.514263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5969.557118] Lustre: lustre-MDT0000: new disk, initializing [ 5969.597126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5969.607655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5971.104498] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5975.602874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5975.625020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5975.662174] Lustre: 394346:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5975.684201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5975.685764] Lustre: Skipped 1 previous similar message [ 5975.722956] Lustre: lustre-MDT0001: new disk, initializing [ 5975.758131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5975.774017] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5975.778894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5977.162661] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5980.502890] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5981.846533] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5984.433925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5984.458548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5984.552018] Lustre: lustre-OST0000: new disk, initializing [ 5984.554272] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5984.586119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5986.687223] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5989.880137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5989.884166] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5989.897818] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5989.921229] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5991.324558] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5991.423220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5992.825449] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5992.902632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5994.981859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5994.982049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5994.987274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5994.989308] LustreError: Skipped 1 previous similar message [ 5999.662102] LustreError: 397287:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5999.704085] Lustre: server umount lustre-OST0000 complete [ 6001.631934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6001.634941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6001.638193] Lustre: Skipped 1 previous similar message [ 6001.640536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6001.643471] Lustre: Skipped 1 previous similar message [ 6005.216257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6005.216848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6007.178468] LustreError: 397489:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6007.182887] LustreError: 397489:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6007.220281] Lustre: server umount lustre-MDT0000 complete [ 6008.765240] LustreError: 395128:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172388 with bad export cookie 13504116119515017089 [ 6008.768469] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6008.770590] LustreError: 395128:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6008.839958] LustreError: 397692:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6008.842019] LustreError: 397692:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6008.906647] Lustre: server umount lustre-MDT0001 complete [ 6012.460738] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 07:19:51 (1762172391) [ 6016.304622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6019.711192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6019.889040] Lustre: lustre-MDT0000: new disk, initializing [ 6019.927757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6021.406686] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6024.589040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6024.702974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6026.069755] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6028.010286] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6029.323959] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6034.912906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6034.913426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6034.914984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6034.914993] Lustre: Skipped 1 previous similar message [ 6034.924353] Lustre: Skipped 1 previous similar message [ 6036.220970] LustreError: 400300:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6036.223495] LustreError: 400300:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6036.253371] Lustre: server umount lustre-MDT0000 complete [ 6037.450223] LustreError: 398938:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172417 with bad export cookie 13504116119515017971 [ 6037.453577] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6037.454261] LustreError: 398938:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6037.567849] Lustre: server umount lustre-MDT0001 complete [ 6040.868632] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 07:20:19 (1762172419) [ 6043.153100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6043.379710] LustreError: 401222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6043.392876] LustreError: 401222:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6043.421329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6044.820849] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6048.011974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6049.321755] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6051.085615] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6052.279373] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6054.478184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6056.601369] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6058.642039] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6077.008612] Lustre: Permanently deactivating lustre-OST0000 [ 6077.013456] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6085.630497] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6086.734042] LustreError: 404032:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6086.738419] LustreError: 404032:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 6086.741249] LustreError: 404032:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6087.352953] LustreError: 404081:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 6087.356637] LustreError: 404081:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6094.189263] LustreError: 404178:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6094.194655] LustreError: 404178:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6094.238641] Lustre: server umount lustre-OST0000 complete [ 6095.278666] LustreError: 404331:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 6095.281952] LustreError: 404331:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6099.424096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6099.426843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6099.430430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6099.431983] Lustre: Skipped 3 previous similar messages [ 6106.084589] LustreError: 404428:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6106.087035] LustreError: 404428:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6106.110868] Lustre: server umount lustre-MDT0000 complete [ 6107.357371] LustreError: 403591:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172487 with bad export cookie 13504116119515018510 [ 6107.360715] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6107.361748] LustreError: 403591:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6117.911653] Lustre: server umount lustre-MDT0001 complete [ 6121.168818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6122.465320] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6123.370481] LustreError: 405473:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6125.478866] Lustre: 405811:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.137@tcp [ 6126.219409] LustreError: 405908:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6126.221977] LustreError: 405908:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6126.247296] Lustre: server umount MGS complete [ 6129.965584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6130.117831] LustreError: 406536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6130.122389] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6130.126621] Lustre: Skipped 1 previous similar message [ 6130.146867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6130.148974] Lustre: Skipped 2 previous similar messages [ 6131.435464] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6134.357298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6134.450163] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6135.857025] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6137.952524] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6139.309994] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6139.960095] Lustre: Permanently reactivating lustre-OST0000 [ 6139.964404] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6139.966757] Lustre: Skipped 2 previous similar messages [ 6165.983210] LustreError: 408131:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6168.336650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6168.450104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6168.452732] Lustre: Skipped 1 previous similar message [ 6170.433628] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6172.192732] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6185.955201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6185.955567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6185.958276] LustreError: Skipped 1 previous similar message [ 6185.963806] Lustre: Skipped 1 previous similar message [ 6185.966481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6185.969776] Lustre: Skipped 5 previous similar messages [ 6190.189082] LustreError: 409917:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6190.191223] LustreError: 409917:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6190.234411] Lustre: server umount lustre-OST0000 complete [ 6196.192508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6197.594548] Lustre: server umount lustre-MDT0000 complete [ 6198.821437] LustreError: 406516:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172578 with bad export cookie 13504116119515019455 [ 6198.824652] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6198.826665] LustreError: 406516:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6198.953445] Lustre: server umount lustre-MDT0001 complete [ 6202.680594] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6203.813743] Key type lgssc unregistered [ 6203.950431] LNet: 410857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6203.954793] LNetError: 410857:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6203.966354] LNet: Removed LNI 192.168.202.137@tcp [ 6204.297128] Key type .llcrypt unregistered [ 6204.298504] Key type ._llcrypt unregistered [ 6210.722418] Key type ._llcrypt registered [ 6210.723498] Key type .llcrypt registered [ 6210.765890] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 6217.150237] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6217.494294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6217.548362] alg: No test for adler32 (adler32-zlib) [ 6218.441393] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 6218.536363] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 6220.127121] Key type lgssc registered [ 6220.549037] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6223.929911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6226.749129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6229.174654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6231.488486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6233.774883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6233.793995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6233.798761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6234.905048] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6234.919609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6234.960545] Lustre: lustre-MDT0000: new disk, initializing [ 6234.991079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6234.999531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6236.503438] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6241.646464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6241.683760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6241.732404] Lustre: 413968:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6241.773478] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6241.775641] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6241.776199] Lustre: Skipped 1 previous similar message [ 6241.815793] Lustre: lustre-MDT0001: new disk, initializing [ 6241.860056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6241.881472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6241.887584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6243.667607] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6247.524327] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6248.946169] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6251.553086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6251.588183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6251.694142] Lustre: lustre-OST0000: new disk, initializing [ 6251.697090] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6251.727962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6253.826949] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6255.490886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6255.497706] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6255.541161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6257.336549] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6259.068264] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6259.152147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6260.714219] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6260.810594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6265.825385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6265.825904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6265.828626] LustreError: Skipped 1 previous similar message [ 6265.833100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6265.836155] Lustre: Skipped 1 previous similar message [ 6267.758070] LustreError: 416905:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6267.802448] Lustre: server umount lustre-OST0000 complete [ 6270.944628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6270.945486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6270.948851] Lustre: Skipped 2 previous similar messages [ 6270.953042] Lustre: Skipped 1 previous similar message [ 6275.270324] LustreError: 417106:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6275.273492] LustreError: 417106:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6275.304027] Lustre: server umount lustre-MDT0000 complete [ 6276.762541] LustreError: 414747:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172656 with bad export cookie 9038914511262456466 [ 6276.764865] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6276.767434] LustreError: 414747:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6276.838321] LustreError: 417308:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6276.840196] LustreError: 417308:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6276.905916] Lustre: server umount lustre-MDT0001 complete [ 6280.099066] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6280.720309] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 07:24:19 (1762172659) [ 6283.205623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6283.363371] LustreError: 418027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6283.370633] LustreError: 418027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6283.390151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6284.822562] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6287.696448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6287.825900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6289.098794] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6290.936972] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6292.347231] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6294.930533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6297.200398] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6299.259454] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6299.980167] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6310.863710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6313.443031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6313.445594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6313.451531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6313.454481] Lustre: Skipped 1 previous similar message [ 6318.447260] LustreError: 420488:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6318.449752] LustreError: 420488:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6318.501214] Lustre: server umount lustre-OST0000 complete [ 6323.681843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6323.682091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6323.684764] LustreError: Skipped 1 previous similar message [ 6323.690437] Lustre: Skipped 1 previous similar message [ 6323.691667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6323.697322] Lustre: Skipped 1 previous similar message [ 6325.890305] LustreError: 420690:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6325.892965] LustreError: 420690:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6325.925380] Lustre: server umount lustre-MDT0000 complete [ 6327.313377] LustreError: 418746:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172706 with bad export cookie 9038914511262457348 [ 6327.315882] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6327.317136] LustreError: 418746:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6327.434852] Lustre: server umount lustre-MDT0001 complete [ 6331.363608] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6332.553343] Key type lgssc unregistered [ 6332.694599] LNet: 421422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6332.697476] LNetError: 421422:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6332.704339] LNet: Removed LNI 192.168.202.137@tcp [ 6333.035844] Key type .llcrypt unregistered [ 6333.037576] Key type ._llcrypt unregistered [ 6341.359289] Key type ._llcrypt registered [ 6341.360432] Key type .llcrypt registered [ 6341.397889] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6341.502493] Key type .llcrypt unregistered [ 6341.503410] Key type ._llcrypt unregistered [ 6348.546943] Key type ._llcrypt registered [ 6348.547827] Key type .llcrypt registered [ 6348.583481] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6348.921981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6348.970076] alg: No test for adler32 (adler32-zlib) [ 6349.830364] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 6349.917338] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 6351.503172] Key type lgssc registered [ 6351.898035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6354.580091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6354.585442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6355.738585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6357.004557] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6359.805780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6359.930660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6362.014268] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6363.990914] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6365.153170] LustreError: 423572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6365.158516] LustreError: 423572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6366.176773] LustreError: 423572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6366.180631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6366.184084] LustreError: 423572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6366.610712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6366.750803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6368.150926] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6374.413767] Lustre: 423567:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 6374.416501] Lustre: 423567:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6374.419198] Lustre: 423567:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 6374.422883] Lustre: 423567:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/153/0, punch: 0/0/0, quota 1/3/0 [ 6374.426828] Lustre: 423567:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6374.429994] Lustre: 423567:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6376.416080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6376.418939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6376.422644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6376.929206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6376.929854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6376.934038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6381.485077] LustreError: 425627:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6381.527038] Lustre: server umount lustre-OST0000 complete [ 6386.145747] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6386.148624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6386.152726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6388.991177] LustreError: 425828:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6388.994126] LustreError: 425828:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6389.022914] Lustre: server umount lustre-MDT0000 complete [ 6390.413492] LustreError: 423553:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172770 with bad export cookie 16337964870969725100 [ 6390.416245] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6390.418755] LustreError: 423553:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6390.498143] LustreError: 426030:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6390.500955] LustreError: 426030:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6390.562226] Lustre: server umount lustre-MDT0001 complete [ 6394.346845] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6395.394939] Key type lgssc unregistered [ 6395.524750] LNet: 426562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6395.528456] LNetError: 426562:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6395.536330] LNet: Removed LNI 192.168.202.137@tcp [ 6395.858543] Key type .llcrypt unregistered [ 6395.860078] Key type ._llcrypt unregistered [ 6405.222679] Key type ._llcrypt registered [ 6405.223727] Key type .llcrypt registered [ 6405.264613] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6405.585070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6405.609353] alg: No test for adler32 (adler32-zlib) [ 6406.464410] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 6406.546409] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 6408.127087] Key type lgssc registered [ 6408.517953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6411.323485] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6411.332179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6426.783320] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6431.903251] LustreError: 427909:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.137@tcp: failed processing log, type 4: rc = -110 [ 6458.527127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6460.643502] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6462.561588] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6464.936184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6465.126712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6465.150256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6466.560737] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6469.707419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6469.879907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6471.371865] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6473.276911] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6474.526334] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6475.807548] Lustre: 428832:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 6475.811906] Lustre: 428832:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6475.814442] Lustre: 428832:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 6475.817024] Lustre: 428832:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 6475.819806] Lustre: 428832:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6475.822330] Lustre: 428832:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6478.305719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6478.307551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6478.310127] Lustre: Skipped 1 previous similar message [ 6482.991108] LustreError: 430186:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 6483.029483] Lustre: server umount lustre-OST0000 complete [ 6485.472035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6485.475568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6485.481667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6485.484204] Lustre: Skipped 1 previous similar message [ 6488.544136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6488.544715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6490.428668] LustreError: 430391:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6490.430452] LustreError: 430391:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6490.459816] Lustre: server umount lustre-MDT0000 complete [ 6491.713018] LustreError: 427916:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172871 with bad export cookie 1144738171576084157 [ 6491.715893] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6491.716991] LustreError: 427916:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6491.778839] LustreError: 430593:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6491.781656] LustreError: 430593:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6491.834695] Lustre: server umount lustre-MDT0001 complete [ 6495.434820] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6496.480833] Key type lgssc unregistered [ 6496.608456] LNet: 431124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6496.611870] LNetError: 431124:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6496.622414] LNet: Removed LNI 192.168.202.137@tcp [ 6496.941341] Key type .llcrypt unregistered [ 6496.942689] Key type ._llcrypt unregistered [ 6506.590738] Key type ._llcrypt registered [ 6506.591861] Key type .llcrypt registered [ 6506.626727] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6506.964713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6507.004457] alg: No test for adler32 (adler32-zlib) [ 6507.863563] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 6507.952662] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 6509.535177] Key type lgssc registered [ 6509.933542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6512.687279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6512.693464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6513.837280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6515.044820] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6517.797081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6517.906185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6519.138294] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6520.820124] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6522.032798] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6524.249235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6524.352221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6525.413254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6526.204405] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6527.925760] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6532.577712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6532.578978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6532.580944] Lustre: Skipped 2 previous similar messages [ 6532.583843] Lustre: Skipped 2 previous similar messages [ 6534.537594] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.37@tcp (stopping) [ 6535.870169] LustreError: 434742:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6535.896024] Lustre: server umount lustre-MDT0000 complete [ 6536.621764] LustreError: 434744:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.37@tcp arrived at 1762172916 with bad export cookie 798924168766438834 [ 6536.626060] LustreError: 434744:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6537.264084] LustreError: 434744:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172916 with bad export cookie 798924168766437952 [ 6537.267654] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6537.696120] LustreError: 433223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6537.696525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6537.696686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6537.697310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6537.701341] LustreError: 433223:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6542.815903] LustreError: 433223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6542.816224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6542.821039] LustreError: 433223:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6543.340065] LustreError: 434946:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6543.341781] LustreError: 434946:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6543.374225] Lustre: server umount lustre-OST0000 complete [ 6545.457899] LustreError: 435299:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6545.459644] LustreError: 435299:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6545.509394] Lustre: server umount lustre-MDT0001 complete [ 6548.875207] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6549.877891] Key type lgssc unregistered [ 6550.008395] LNet: 435829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6550.011706] LNetError: 435829:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6550.028300] LNet: Removed LNI 192.168.202.137@tcp [ 6550.345275] Key type .llcrypt unregistered [ 6550.346333] Key type ._llcrypt unregistered [ 6559.190879] Key type ._llcrypt registered [ 6559.191905] Key type .llcrypt registered [ 6559.234879] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6559.659615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6559.665971] alg: No test for adler32 (adler32-zlib) [ 6560.516496] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 6560.596876] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 6562.175130] Key type lgssc registered [ 6562.522558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6565.056545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6565.061465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6566.206721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6567.548231] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6570.419548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6570.564765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6571.882252] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6573.632215] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6574.813820] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6577.058888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6577.166555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6579.001428] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6580.688129] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6591.522207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6601.183820] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6601.186802] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6601.191786] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6601.697877] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6601.701363] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6601.702783] Lustre: Skipped 1 previous similar message [ 6603.450088] LustreError: 439470:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6603.474460] Lustre: server umount lustre-MDT0001 complete [ 6606.816492] LustreError: 437204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6606.816633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6606.821956] LustreError: 437204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6606.824263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6606.829797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6611.246098] LustreError: 439675:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6611.248017] LustreError: 439675:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6611.280832] Lustre: server umount lustre-OST0000 complete [ 6611.937624] LustreError: 437204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6612.519567] LustreError: 439877:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6612.521333] LustreError: 439877:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6612.583634] Lustre: server umount lustre-MDT0000 complete [ 6616.676863] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6617.640104] Key type lgssc unregistered [ 6617.765314] LNet: 440560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6617.767497] LNetError: 440560:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6617.774295] LNet: Removed LNI 192.168.202.137@tcp [ 6618.074922] Key type .llcrypt unregistered [ 6618.076228] Key type ._llcrypt unregistered [ 6625.477690] Key type ._llcrypt registered [ 6625.478603] Key type .llcrypt registered [ 6625.513069] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 6631.105870] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6631.515882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6631.525995] alg: No test for adler32 (adler32-zlib) [ 6632.367021] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 6632.437478] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 6634.007134] Key type lgssc registered [ 6634.335492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6637.220472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6639.412048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6641.526612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6643.648799] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6646.852933] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6649.998508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6652.315556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6652.343080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6652.347595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6653.438766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6653.450357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6653.486579] Lustre: lustre-MDT0000: new disk, initializing [ 6653.511072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6653.520053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6654.857505] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6658.363016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6660.588257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6660.610834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6660.701695] Lustre: lustre-OST0000: new disk, initializing [ 6660.703718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6660.705314] Lustre: Skipped 1 previous similar message [ 6660.730724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6661.503613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6661.506721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6661.514793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6662.566859] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6665.581530] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6668.837878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6671.251251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6671.283094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6671.323752] Lustre: 444504:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6671.366195] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6671.408506] Lustre: lustre-MDT0001: new disk, initializing [ 6671.461898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6671.478829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6671.483411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6672.889356] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6676.600310] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6676.686332] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6678.113377] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6679.212925] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 6683.618441] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6683.619875] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6683.623092] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6683.628017] Lustre: Skipped 1 previous similar message [ 6685.067025] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.37@tcp (stopping) [ 6688.737708] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6688.739632] Lustre: Skipped 1 previous similar message [ 6693.857710] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6693.860534] Lustre: Skipped 2 previous similar messages [ 6695.391190] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6695.433943] LustreError: 447934:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6695.459960] Lustre: server umount lustre-MDT0001 complete [ 6704.097989] LustreError: 444511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6704.103307] LustreError: 444511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6709.217829] LustreError: 444515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.218130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6709.227139] Lustre: Skipped 1 previous similar message [ 6709.228963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6709.230507] Lustre: Skipped 1 previous similar message [ 6714.336851] LustreError: 444515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6719.457201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6719.458391] LustreError: 444515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6719.459519] Lustre: Skipped 1 previous similar message [ 6722.015090] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6722.032734] LustreError: 448206:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6722.035204] LustreError: 448206:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6722.072533] Lustre: server umount lustre-OST0000 complete [ 6729.406346] LustreError: 448409:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6729.408268] LustreError: 448409:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6729.462744] Lustre: server umount lustre-MDT0000 complete [ 6734.147432] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6735.208087] Key type lgssc unregistered [ 6735.340360] LNet: 449093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6735.342726] LNetError: 449093:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6735.352330] LNet: Removed LNI 192.168.202.137@tcp [ 6735.665953] Key type .llcrypt unregistered [ 6735.667558] Key type ._llcrypt unregistered [ 6749.312974] Key type ._llcrypt registered [ 6749.314239] Key type .llcrypt registered [ 6749.355334] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6749.466245] Key type .llcrypt unregistered [ 6749.467769] Key type ._llcrypt unregistered [ 6756.119689] Key type ._llcrypt registered [ 6756.120755] Key type .llcrypt registered [ 6756.160867] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6756.588170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6756.618621] alg: No test for adler32 (adler32-zlib) [ 6757.473515] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 6757.555885] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 6759.135163] Key type lgssc registered [ 6759.496360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6762.108760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6764.322248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6766.344689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6768.693430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6768.718086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6768.723650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6769.809299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6769.822130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6769.859223] Lustre: lustre-MDT0000: new disk, initializing [ 6769.882370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6769.888589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6771.206546] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6775.391561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6775.419179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6775.450027] Lustre: 452411:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6775.472756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6775.474510] Lustre: Skipped 1 previous similar message [ 6775.510684] Lustre: lustre-MDT0001: new disk, initializing [ 6775.543910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6775.556207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6775.559487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6776.825780] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6779.930760] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6781.150212] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6783.411700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6783.444144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6783.542566] Lustre: lustre-OST0000: new disk, initializing [ 6783.544858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6783.573769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6785.493043] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6788.087840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6788.091446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6788.126648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6788.737156] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6791.135819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6791.139259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6791.144094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6793.185792] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6793.186039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6793.189976] Lustre: Skipped 1 previous similar message [ 6793.195473] Lustre: Skipped 2 previous similar messages [ 6797.248374] LustreError: 454852:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6797.283335] Lustre: server umount lustre-MDT0000 complete [ 6798.494119] LustreError: 452402:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762173178 with bad export cookie 5673870900125228092 [ 6798.497626] LustreError: 452402:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6798.497826] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6798.568735] LustreError: 455053:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6798.571841] LustreError: 455053:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6798.622982] Lustre: server umount lustre-MDT0001 complete [ 6810.097276] LustreError: 455253:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6810.098782] LustreError: 455253:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6810.148384] Lustre: server umount lustre-OST0000 complete [ 6813.630481] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 07:33:12 (1762173192) [ 6813.997148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6816.269188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6816.444371] LustreError: 456082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6816.449094] LustreError: 456082:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6816.468543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6817.789667] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6820.562294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6820.688162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6821.836432] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6823.606936] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6824.783589] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6827.050333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6827.074380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6827.137616] Lustre: Found index 0 for lustre-OST0000, updating log [ 6829.103668] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6830.987503] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6841.576826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6846.945331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6846.945619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6846.948635] LustreError: Skipped 1 previous similar message [ 6846.955180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6846.957612] Lustre: Skipped 1 previous similar message [ 6849.197869] LustreError: 458359:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6849.199549] LustreError: 458359:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6849.231170] Lustre: server umount lustre-OST0000 complete [ 6850.181320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6851.551971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6851.555236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6851.558454] Lustre: Skipped 1 previous similar message [ 6851.560226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6851.561978] Lustre: Skipped 2 previous similar messages [ 6859.232315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6859.234467] Lustre: Skipped 5 previous similar messages [ 6865.375130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6865.409108] LustreError: 458613:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6865.411525] LustreError: 458613:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6865.437455] Lustre: server umount lustre-MDT0000 complete [ 6866.700182] LustreError: 456063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762173246 with bad export cookie 5673870900125229478 [ 6866.703139] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6866.704014] LustreError: 456063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6866.811032] Lustre: server umount lustre-MDT0001 complete [ 6869.637082] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 07:34:08 (1762173248) [ 6869.978653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6870.308306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6870.635709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6870.970297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6871.325047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6871.676025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6872.010315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6872.326627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6872.662901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6873.006677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6873.336279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6873.660938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6873.990721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6874.326247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6874.691685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6875.040047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6875.384435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6875.739593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6876.090368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6876.460765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6876.814337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6877.143711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6877.497328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6877.846324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6878.189626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6878.520231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6878.839278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6879.158919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6879.504758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6879.859369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6880.191561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6880.548088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6880.914064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6881.238606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6881.558261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6881.888491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6882.218165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6882.570380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6882.899806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6883.213331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6883.550425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6883.889170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6884.220875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6884.574825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6884.914735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6885.258033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6885.610298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6885.948679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6886.268066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6886.597644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6886.916815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6889.501558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6889.656885] LustreError: 462195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6889.682986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6889.684782] Lustre: Skipped 1 previous similar message [ 6891.058873] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6893.963226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6895.360780] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6897.300827] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6898.435615] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6900.752801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6900.784694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6900.870553] Lustre: Found index 0 for lustre-OST0000, updating log [ 6901.996938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6902.845882] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6904.643229] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6907.359744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6907.361625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6907.364261] Lustre: Skipped 1 previous similar message [ 6907.365630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6907.366959] Lustre: Skipped 2 previous similar messages [ 6911.339168] LustreError: 464509:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6911.342284] LustreError: 464509:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6911.372850] Lustre: server umount lustre-OST0000 complete [ 6918.678863] Lustre: server umount lustre-MDT0000 complete [ 6918.689305] LustreError: 462907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6918.694962] LustreError: 462907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6919.889936] LustreError: 462175:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762173299 with bad export cookie 5673870900125230549 [ 6919.892720] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6919.895575] LustreError: 462175:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6920.031519] Lustre: server umount lustre-MDT0001 complete [ 6920.997351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6926.006499] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 6928.920186] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6932.343146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6934.422889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6936.515751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6938.673747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6941.040786] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 07:35:19 (1762173319) [ 6964.570863] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6965.667775] Key type lgssc unregistered [ 6965.819653] LNet: 468494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6965.822245] LNetError: 468494:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7091.814682] LNet: Removed LNI 192.168.202.137@tcp [ 7092.149565] Key type .llcrypt unregistered [ 7092.150926] Key type ._llcrypt unregistered [ 7098.716686] Key type ._llcrypt registered [ 7098.717696] Key type .llcrypt registered [ 7098.756408] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7099.109712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7099.157518] alg: No test for adler32 (adler32-zlib) [ 7100.014088] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 7100.107072] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 7101.695098] Key type lgssc registered [ 7102.052252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7103.517980] LNet: Added LNI 192.168.202.137@tcp100 [8/256/0/180] [ 7103.821791] LNet: Added LNI 192.168.202.137@tcp101 [8/256/0/180] [ 7104.109288] LNet: Added LNI 192.168.202.137@tcp102 [8/256/0/180] [ 7104.404651] LNet: Added LNI 192.168.202.137@tcp103 [8/256/0/180] [ 7104.685980] LNet: Added LNI 192.168.202.137@tcp104 [8/256/0/180] [ 7104.969761] LNet: Added LNI 192.168.202.137@tcp105 [8/256/0/180] [ 7105.250559] LNet: Added LNI 192.168.202.137@tcp106 [8/256/0/180] [ 7105.525349] LNet: Added LNI 192.168.202.137@tcp107 [8/256/0/180] [ 7105.798457] LNet: Added LNI 192.168.202.137@tcp108 [8/256/0/180] [ 7106.075851] LNet: Added LNI 192.168.202.137@tcp109 [8/256/0/180] [ 7106.372734] LNet: Added LNI 192.168.202.137@tcp110 [8/256/0/180] [ 7106.669646] LNet: Added LNI 192.168.202.137@tcp111 [8/256/0/180] [ 7106.977705] LNet: Added LNI 192.168.202.137@tcp112 [8/256/0/180] [ 7107.285197] LNet: Added LNI 192.168.202.137@tcp113 [8/256/0/180] [ 7107.608200] LNet: Added LNI 192.168.202.137@tcp114 [8/256/0/180] [ 7107.907377] LNet: Added LNI 192.168.202.137@tcp115 [8/256/0/180] [ 7108.230851] LNet: Added LNI 192.168.202.137@tcp116 [8/256/0/180] [ 7108.548036] LNet: Added LNI 192.168.202.137@tcp117 [8/256/0/180] [ 7108.850522] LNet: Added LNI 192.168.202.137@tcp118 [8/256/0/180] [ 7109.157763] LNet: Added LNI 192.168.202.137@tcp119 [8/256/0/180] [ 7109.461245] LNet: Added LNI 192.168.202.137@tcp120 [8/256/0/180] [ 7109.765385] LNet: Added LNI 192.168.202.137@tcp121 [8/256/0/180] [ 7110.071419] LNet: Added LNI 192.168.202.137@tcp122 [8/256/0/180] [ 7110.373988] LNet: Added LNI 192.168.202.137@tcp123 [8/256/0/180] [ 7110.668930] LNet: Added LNI 192.168.202.137@tcp124 [8/256/0/180] [ 7110.968819] LNet: Added LNI 192.168.202.137@tcp125 [8/256/0/180] [ 7111.283308] LNet: Added LNI 192.168.202.137@tcp126 [8/256/0/180] [ 7111.584131] LNet: Added LNI 192.168.202.137@tcp127 [8/256/0/180] [ 7111.870135] LNet: Added LNI 192.168.202.137@tcp128 [8/256/0/180] [ 7112.156343] LNet: Added LNI 192.168.202.137@tcp129 [8/256/0/180] [ 7112.458464] LNet: Added LNI 192.168.202.137@tcp130 [8/256/0/180] [ 7112.766957] LNet: Added LNI 192.168.202.137@tcp131 [8/256/0/180] [ 7113.058327] LNet: Added LNI 192.168.202.137@tcp132 [8/256/0/180] [ 7113.350749] LNet: Added LNI 192.168.202.137@tcp133 [8/256/0/180] [ 7113.634826] LNet: Added LNI 192.168.202.137@tcp134 [8/256/0/180] [ 7113.937302] LNet: Added LNI 192.168.202.137@tcp135 [8/256/0/180] [ 7116.231583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7116.256504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7116.262816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7117.315257] Lustre: 471483:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.137@tcp: can't add NID 192.168.202.137@tcp130: rc = -75 [ 7117.380209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7117.390142] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7117.430421] Lustre: lustre-MDT0000: new disk, initializing [ 7117.459101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7117.465299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7118.712514] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7122.701955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7122.723910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7122.767434] Lustre: 471503:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7122.794319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7122.795951] Lustre: Skipped 1 previous similar message [ 7122.832708] Lustre: lustre-MDT0001: new disk, initializing [ 7122.843692] Lustre: 472288:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.137@tcp: can't add NID 192.168.202.137@tcp130: rc = -75 [ 7122.846263] Lustre: 472288:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7122.883477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7122.895140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7122.899764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7124.149160] Lustre: 472503:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.137@tcp: can't add NID 192.168.202.137@tcp130: rc = -75 [ 7124.153319] Lustre: 472503:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7124.220829] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7127.297925] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7128.471684] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7130.798526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7130.825239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7130.935331] Lustre: lustre-OST0000: new disk, initializing [ 7130.938353] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7130.969876] Lustre: 473257:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.137@tcp: can't add NID 192.168.202.137@tcp130: rc = -75 [ 7130.972389] Lustre: 473257:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 7130.987414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7132.099230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7132.103781] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7132.142983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7132.789780] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7136.033706] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7142.369947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7142.370315] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7142.372088] LustreError: Skipped 1 previous similar message [ 7142.376464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7142.379669] Lustre: Skipped 1 previous similar message [ 7144.428091] LustreError: 474090:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7144.463636] Lustre: server umount lustre-OST0000 complete [ 7147.490067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7147.490685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7147.493667] Lustre: Skipped 2 previous similar messages [ 7147.497902] Lustre: Skipped 1 previous similar message [ 7151.805506] LustreError: 474292:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7151.807413] LustreError: 474292:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7151.833678] Lustre: server umount lustre-MDT0000 complete [ 7153.048074] LustreError: 471494:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762173532 with bad export cookie 17086486301712112944 [ 7153.050515] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7153.051617] LustreError: 471494:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7153.127662] LustreError: 474494:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7153.129842] LustreError: 474494:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7153.186643] Lustre: server umount lustre-MDT0001 complete [ 7157.024089] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 7158.038303] Key type lgssc unregistered [ 7158.191408] LNet: 475033:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7158.194322] LNetError: 475033:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7158.196553] LNet: Removed LNI 192.168.202.137@tcp [ 7159.199131] LNet: Removed LNI 192.168.202.137@tcp100 [ 7159.201116] LNetError: 475033:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7159.203213] LNetError: 475033:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7159.205376] LNet: Removed LNI 192.168.202.137@tcp101 [ 7160.223150] LNet: Removed LNI 192.168.202.137@tcp102 [ 7160.224461] LNetError: 475033:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7160.227462] LNetError: 475033:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7160.230084] LNet: Removed LNI 192.168.202.137@tcp103 [ 7161.247195] LNet: Removed LNI 192.168.202.137@tcp104 [ 7162.271132] LNet: Removed LNI 192.168.202.137@tcp105 [ 7162.273187] LNetError: 475033:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7162.275426] LNetError: 475033:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7162.278280] LNet: Removed LNI 192.168.202.137@tcp106 [ 7163.295134] LNet: Removed LNI 192.168.202.137@tcp107 [ 7164.319158] LNet: Removed LNI 192.168.202.137@tcp108 [ 7165.343185] LNet: Removed LNI 192.168.202.137@tcp109 [ 7166.367166] LNet: Removed LNI 192.168.202.137@tcp110 [ 7166.369342] LNetError: 475033:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7166.373266] LNetError: 475033:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7166.377596] LNet: Removed LNI 192.168.202.137@tcp111 [ 7166.379906] LNet: Removed LNI 192.168.202.137@tcp112 [ 7166.381385] LNet: Removed LNI 192.168.202.137@tcp113 [ 7166.382583] LNet: Removed LNI 192.168.202.137@tcp114 [ 7166.384534] LNet: Removed LNI 192.168.202.137@tcp115 [ 7166.386128] LNet: Removed LNI 192.168.202.137@tcp116 [ 7166.388192] LNet: Removed LNI 192.168.202.137@tcp117 [ 7166.389463] LNet: Removed LNI 192.168.202.137@tcp118 [ 7166.390586] LNet: Removed LNI 192.168.202.137@tcp119 [ 7166.391828] LNet: Removed LNI 192.168.202.137@tcp120 [ 7166.393963] LNet: Removed LNI 192.168.202.137@tcp121 [ 7166.396230] LNet: Removed LNI 192.168.202.137@tcp122 [ 7166.398626] LNet: Removed LNI 192.168.202.137@tcp123 [ 7166.400804] LNet: Removed LNI 192.168.202.137@tcp124 [ 7166.402870] LNet: Removed LNI 192.168.202.137@tcp125 [ 7166.405037] LNet: Removed LNI 192.168.202.137@tcp126 [ 7166.407066] LNet: Removed LNI 192.168.202.137@tcp127 [ 7166.409209] LNet: Removed LNI 192.168.202.137@tcp128 [ 7166.411153] LNet: Removed LNI 192.168.202.137@tcp129 [ 7166.413045] LNet: Removed LNI 192.168.202.137@tcp130 [ 7166.414093] LNet: Removed LNI 192.168.202.137@tcp131 [ 7166.415911] LNet: Removed LNI 192.168.202.137@tcp132 [ 7166.416972] LNet: Removed LNI 192.168.202.137@tcp133 [ 7166.418043] LNet: Removed LNI 192.168.202.137@tcp134 [ 7166.426410] LNet: Removed LNI 192.168.202.137@tcp135 [ 7166.693845] Key type .llcrypt unregistered [ 7166.695143] Key type ._llcrypt unregistered [ 7174.017337] Key type ._llcrypt registered [ 7174.018319] Key type .llcrypt registered [ 7174.055603] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 7179.976194] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7180.312962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7180.336070] alg: No test for adler32 (adler32-zlib) [ 7181.180327] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 7181.258834] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 7182.839218] Key type lgssc registered [ 7183.343381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7186.249988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7188.651496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7190.917553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7192.973633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7195.161913] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 07:39:34 (1762173574) [ 7201.066096] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 7202.098695] Key type lgssc unregistered [ 7202.221421] LNet: 479045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7202.224072] LNetError: 479045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7202.230340] LNet: Removed LNI 192.168.202.137@tcp [ 7202.499510] Key type .llcrypt unregistered [ 7202.500454] Key type ._llcrypt unregistered [ 7209.422688] Key type ._llcrypt registered [ 7209.424222] Key type .llcrypt registered [ 7209.462344] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7209.802949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7209.841308] alg: No test for adler32 (adler32-zlib) [ 7210.691752] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 7210.774666] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 7212.359229] Key type lgssc registered [ 7212.766821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7216.181896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7216.215214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7216.223885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7217.319641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7217.330354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7217.368062] Lustre: lustre-MDT0000: new disk, initializing [ 7217.391345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7217.397476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7218.653359] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7223.452021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7223.478138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7223.509788] Lustre: 480216:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7223.532118] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7223.534761] Lustre: Skipped 1 previous similar message [ 7223.570120] Lustre: lustre-MDT0001: new disk, initializing [ 7223.604137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7223.615318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7223.618097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7224.941843] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7228.179479] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7229.434049] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7231.844137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7231.870033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7231.961376] Lustre: lustre-OST0000: new disk, initializing [ 7231.963426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7231.989432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7232.233400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7232.237071] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7232.269186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7233.959693] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7237.019627] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7238.119694] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing lnet_if_list [ 7239.850091] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7239.861166] LNet: Added LNI 192.168.202.137@tcp42 [8/256/0/180] [ 7247.843781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7247.844359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7247.847910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7247.851899] Lustre: Skipped 1 previous similar message [ 7252.652981] LustreError: 483602:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7252.688546] Lustre: server umount lustre-OST0000 complete [ 7254.496566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7254.501141] LustreError: Skipped 1 previous similar message [ 7254.503879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7254.513927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7254.517258] Lustre: Skipped 1 previous similar message [ 7258.080356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7258.080800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7260.288090] LustreError: 483803:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7260.290202] LustreError: 483803:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7260.316481] Lustre: server umount lustre-MDT0000 complete [ 7261.521084] LustreError: 480995:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762173641 with bad export cookie 9874595962153214989 [ 7261.524163] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7261.526824] LustreError: 480995:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7261.587233] LustreError: 484005:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7261.588857] LustreError: 484005:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7261.635470] Lustre: server umount lustre-MDT0001 complete [ 7265.030558] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 7266.089486] Key type lgssc unregistered [ 7266.252834] LNet: 484543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7266.255053] LNetError: 484543:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7266.257121] LNet: Removed LNI 192.168.202.137@tcp [ 7266.276323] LNet: Removed LNI 192.168.202.137@tcp42 [ 7266.548593] Key type .llcrypt unregistered [ 7266.549574] Key type ._llcrypt unregistered [ 7273.682962] Key type ._llcrypt registered [ 7273.684069] Key type .llcrypt registered [ 7273.715785] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7274.061199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7274.083482] alg: No test for adler32 (adler32-zlib) [ 7274.929463] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 7275.005476] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 7276.583159] Key type lgssc registered [ 7276.931127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7279.431350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7279.437309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7280.586495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7281.806391] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7284.523091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7284.641062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7285.960802] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7287.756959] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7288.927102] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7291.173748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7291.297080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7293.246964] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7294.959467] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7314.498820] loop0: detected capacity change from 0 to 204800000 [ 7314.614902] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7314.619568] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7314.624767] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7314.647210] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7316.761418] loop0: detected capacity change from 0 to 204800000 [ 7316.880092] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7316.883579] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7316.888403] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7316.948318] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7318.729270] loop0: detected capacity change from 0 to 204800000 [ 7320.637169] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7320.655854] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7320.700600] Lustre: fs15246-OST0000: new disk, initializing [ 7320.702485] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7320.719709] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7322.626704] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7326.457861] loop1: detected capacity change from 0 to 204800000 [ 7328.245457] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7328.270547] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7328.296837] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7328.315686] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7328.350091] Lustre: fs15246-MDT0000: new disk, initializing [ 7328.397356] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7328.405292] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7328.437803] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7329.503780] Lustre: 490158:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000006f5b91d5 x1847773001450496/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1762173725 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7329.648289] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7338.591229] LustreError: 490158:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7338.594313] LustreError: 490158:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 7338.596891] LustreError: 490158:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 7340.518275] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7340.524274] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7343.294750] LustreError: 491281:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 7343.321422] Lustre: server umount fs15246-MDT0000 complete [ 7345.637268] LustreError: 485758:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7345.645959] LustreError: 485758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7346.896345] LustreError: 491784:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 7346.898661] LustreError: 491784:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7346.917219] Lustre: server umount fs15246-OST0000 complete [ 7350.755846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7350.757663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7350.763523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7350.772906] Lustre: Skipped 1 previous similar message [ 7355.874195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7355.875907] Lustre: Skipped 1 previous similar message [ 7356.396189] LustreError: 492286:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7356.399019] LustreError: 492286:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7356.438039] Lustre: server umount lustre-OST0000 complete [ 7360.992954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7360.993593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7360.998612] Lustre: Skipped 2 previous similar messages [ 7363.775093] LustreError: 492488:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7363.776696] LustreError: 492488:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7363.802069] Lustre: server umount lustre-MDT0000 complete [ 7365.139162] LustreError: 492199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762173744 with bad export cookie 40670623410853707 [ 7365.142162] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7365.144729] LustreError: 492199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7365.258938] Lustre: server umount lustre-MDT0001 complete [ 7368.798452] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 7369.732644] Key type lgssc unregistered [ 7369.879609] LNet: 493220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7369.883878] LNetError: 493220:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7369.893470] LNet: Removed LNI 192.168.202.137@tcp [ 7370.294503] Key type .llcrypt unregistered [ 7370.296247] Key type ._llcrypt unregistered [ 7377.346748] Key type ._llcrypt registered [ 7377.347846] Key type .llcrypt registered [ 7377.382267] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7377.702332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7377.712276] alg: No test for adler32 (adler32-zlib) [ 7378.557484] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 7378.640667] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 7380.231095] Key type lgssc registered [ 7380.566130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7383.039427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7385.033275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7387.112585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7388.808855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7394.331070] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 7398.289187] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7401.983060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7404.662516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7407.288726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7410.166865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7413.070048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7413.091509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7413.097780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7414.178440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7414.189020] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7414.222505] Lustre: lustre-MDT0000: new disk, initializing [ 7414.244943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7414.250602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7415.540613] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7419.821556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7419.839186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7419.870647] Lustre: 497626:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7419.896385] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7419.898367] Lustre: Skipped 1 previous similar message [ 7419.936224] Lustre: lustre-MDT0001: new disk, initializing [ 7419.967965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7419.980061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7419.982953] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7421.351114] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7424.607396] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7425.885992] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7428.887788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7428.928660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7429.073505] Lustre: lustre-OST0000: new disk, initializing [ 7429.075762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7429.116207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7431.339411] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7434.750198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7434.754090] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7434.799065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7434.966503] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7436.428393] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7436.512006] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7438.006589] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7438.076558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7439.846770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7439.846976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7439.852299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7439.864314] Lustre: Skipped 1 previous similar message [ 7444.849737] LustreError: 500562:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7444.914245] Lustre: server umount lustre-OST0000 complete [ 7450.080657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7450.081365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7450.083806] Lustre: Skipped 1 previous similar message [ 7450.085320] Lustre: Skipped 1 previous similar message [ 7452.795761] LustreError: 500763:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7452.797365] LustreError: 500763:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7452.824137] Lustre: server umount lustre-MDT0000 complete [ 7454.091671] LustreError: 498406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762173833 with bad export cookie 11621063241585779388 [ 7454.094803] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7454.096586] LustreError: 498406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7454.194310] LustreError: 500965:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7454.198610] LustreError: 500965:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7454.271587] Lustre: server umount lustre-MDT0001 complete [ 7463.444496] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 7467.995248] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7471.315402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7473.879992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7476.241806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7479.532195] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7480.222262] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7291 sec ========== 07:44:18 (1762173858) [ 7481.003572] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:44:19 (1762173859) === [ 7482.324073] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:44:21 (1762173861) === [ 7499.905706] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 7501.156360] Key type lgssc unregistered [ 7501.313610] LNet: 505815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7501.318683] LNetError: 505815:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7501.331639] LNet: Removed LNI 192.168.202.137@tcp [ 7501.693929] Key type .llcrypt unregistered [ 7501.694973] Key type ._llcrypt unregistered