[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474916415 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003266] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.007762] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010131] LSM: Security Framework initializing [ 0.011064] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.013058] *** VALIDATE selinux *** [ 0.021168] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025762] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026128] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027132] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028123] *** VALIDATE tmpfs *** [ 0.030261] *** VALIDATE proc *** [ 0.031261] *** VALIDATE cgroup *** [ 0.032000] *** VALIDATE cgroup2 *** [ 0.032286] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033170] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035029] Spectre V2 : User space: Vulnerable [ 0.036010] Speculative Store Bypass: Vulnerable [ 0.039252] debug: unmapping init [mem 0xffffffff84859000-0xffffffff84860fff] [ 0.041165] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042673] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043031] ... version: 2 [ 0.044013] ... bit width: 48 [ 0.045015] ... generic registers: 4 [ 0.046012] ... value mask: 0000ffffffffffff [ 0.047016] ... max period: 00007fffffffffff [ 0.048016] ... fixed-purpose events: 3 [ 0.049012] ... event mask: 000000070000000f [ 0.050319] rcu: Hierarchical SRCU implementation. [ 0.052427] smp: Bringing up secondary CPUs ... [ 0.053576] x86: Booting SMP configuration: [ 0.054025] .... node #0, CPUs: #1 #2 #3 [ 0.057131] smp: Brought up 1 node, 4 CPUs [ 0.059013] smpboot: Max logical packages: 1 [ 0.060015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.149246] node 0 deferred pages initialised in 86ms [ 0.152134] devtmpfs: initialized [ 0.153240] x86/mm: Memory block size: 128MB [ 0.155257] gcov: version magic: 0x41383552 [ 0.157029] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158104] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159246] pinctrl core: initialized pinctrl subsystem [ 0.160226] [ 0.160881] ************************************************************* [ 0.161012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162011] ** ** [ 0.163012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164014] ** ** [ 0.165009] ** This means that this kernel is built to expose internal ** [ 0.166011] ** IOMMU data structures, which may compromise security on ** [ 0.167012] ** your system. ** [ 0.168014] ** ** [ 0.169013] ** If you see this message and you are not debugging the ** [ 0.170014] ** kernel, report this immediately to your vendor! ** [ 0.171014] ** ** [ 0.172013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173013] ************************************************************* [ 0.174616] NET: Registered protocol family 16 [ 0.175411] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.176059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.177067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.178443] cpuidle: using governor menu [ 0.180254] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.185586] PCI: Using configuration type 1 for base access [ 0.187122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.196167] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.197035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198162] cryptd: max_cpu_qlen set to 1000 [ 0.200268] ACPI: Added _OSI(Module Device) [ 0.201028] ACPI: Added _OSI(Processor Device) [ 0.202018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.204013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.208150] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.215391] ACPI: Interpreter enabled [ 0.216055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218015] ACPI: Using IOAPIC for interrupt routing [ 0.219102] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222401] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.232030] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.234045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.236020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.240083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.244191] acpiphp: Slot [2] registered [ 0.246113] acpiphp: Slot [3] registered [ 0.247074] acpiphp: Slot [4] registered [ 0.248089] acpiphp: Slot [5] registered [ 0.250121] acpiphp: Slot [6] registered [ 0.252094] acpiphp: Slot [7] registered [ 0.253097] acpiphp: Slot [8] registered [ 0.255102] acpiphp: Slot [9] registered [ 0.256115] acpiphp: Slot [10] registered [ 0.258125] acpiphp: Slot [11] registered [ 0.259089] acpiphp: Slot [12] registered [ 0.261099] acpiphp: Slot [13] registered [ 0.262112] acpiphp: Slot [14] registered [ 0.264093] acpiphp: Slot [15] registered [ 0.265093] acpiphp: Slot [16] registered [ 0.266065] acpiphp: Slot [17] registered [ 0.268111] acpiphp: Slot [18] registered [ 0.269091] acpiphp: Slot [19] registered [ 0.270076] acpiphp: Slot [20] registered [ 0.272090] acpiphp: Slot [21] registered [ 0.273096] acpiphp: Slot [22] registered [ 0.275103] acpiphp: Slot [23] registered [ 0.276123] acpiphp: Slot [24] registered [ 0.277103] acpiphp: Slot [25] registered [ 0.279102] acpiphp: Slot [26] registered [ 0.280108] acpiphp: Slot [27] registered [ 0.282110] acpiphp: Slot [28] registered [ 0.283104] acpiphp: Slot [29] registered [ 0.285105] acpiphp: Slot [30] registered [ 0.287132] acpiphp: Slot [31] registered [ 0.288063] PCI host bridge to bus 0000:00 [ 0.289032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.292035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.294022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.296026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.299023] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.301023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.303175] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.307278] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.311415] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.320776] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.326056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.328020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.331021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.334019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.337136] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.339786] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.343049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.346650] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.351014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.363817] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.369015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.374892] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.382016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.389016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.401018] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.411548] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.418018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.423016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.439025] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.454070] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.462017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.468016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.481024] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.490694] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.497013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.503017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.513019] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.522000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.528014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.534017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.546017] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.552313] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.558023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.564012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.574014] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.585588] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.588434] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.590394] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.593380] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.595213] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.600110] iommu: Default domain type: Passthrough [ 0.601338] SCSI subsystem initialized [ 0.603144] ACPI: bus type USB registered [ 0.605094] usbcore: registered new interface driver usbfs [ 0.607076] usbcore: registered new interface driver hub [ 0.609090] usbcore: registered new device driver usb [ 0.611144] pps_core: LinuxPPS API ver. 1 registered [ 0.612009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.615060] PTP clock support registered [ 0.618037] EDAC MC: Ver: 3.0.0 [ 0.621131] PCI: Using ACPI for IRQ routing [ 0.622702] NetLabel: Initializing [ 0.624011] NetLabel: domain hash size = 128 [ 0.626013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.627077] NetLabel: unlabeled traffic allowed by default [ 0.630041] vgaarb: loaded [ 0.631262] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.634013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.640000] clocksource: Switched to clocksource kvm-clock [ 0.743923] VFS: Disk quotas dquot_6.6.0 [ 0.745680] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.748326] *** VALIDATE ramfs *** [ 0.749284] *** VALIDATE hugetlbfs *** [ 0.750761] pnp: PnP ACPI init [ 0.753153] pnp: PnP ACPI: found 6 devices [ 0.773128] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.776650] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.778825] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.781050] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.783608] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.785588] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.788470] NET: Registered protocol family 2 [ 0.791123] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.795805] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.799352] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.804103] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.807252] TCP: Hash tables configured (established 65536 bind 65536) [ 0.809649] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.812621] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.815322] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.818289] NET: Registered protocol family 1 [ 0.820660] RPC: Registered named UNIX socket transport module. [ 0.822275] RPC: Registered udp transport module. [ 0.823973] RPC: Registered tcp transport module. [ 0.825539] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.827360] NET: Registered protocol family 44 [ 0.828719] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.830543] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.832586] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.834427] PCI: CLS 0 bytes, default 64 [ 0.835926] Unpacking initramfs... [ 2.210837] debug: unmapping init [mem 0xffff98fcbcc54000-0xffff98fcbffbffff] [ 2.214242] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.216591] software IO TLB: mapped [mem 0x00000000a2c00000-0x00000000a6c00000] (64MB) [ 2.219417] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.700931] Initialise system trusted keyrings [ 2.702241] Key type blacklist registered [ 2.704156] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.712432] zbud: loaded [ 2.715346] *** VALIDATE nfs *** [ 2.716657] *** VALIDATE nfs4 *** [ 2.718268] pstore: using deflate compression [ 2.721761] Platform Keyring initialized [ 2.819193] NET: Registered protocol family 38 [ 2.820832] Key type asymmetric registered [ 2.822204] Asymmetric key parser 'x509' registered [ 2.824102] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.827115] io scheduler mq-deadline registered [ 2.828744] io scheduler kyber registered [ 2.830264] io scheduler bfq registered [ 2.832069] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.834748] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.837485] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.840269] ACPI: Power Button [PWRF] [ 2.933383] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.031351] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.217248] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.315712] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.504861] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.539533] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.573320] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.579792] Non-volatile memory driver v1.3 [ 3.581055] Linux agpgart interface v0.103 [ 3.610583] virtio_blk virtio1: [vda] 133848 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.613555] vda: detected capacity change from 0 to 68530176 [ 3.628464] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.631633] vdb: detected capacity change from 0 to 1073741824 [ 3.646722] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.649936] vdc: detected capacity change from 0 to 2621440000 [ 3.664438] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.667603] vdd: detected capacity change from 0 to 2621440000 [ 3.686380] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.688881] vde: detected capacity change from 0 to 4294967296 [ 3.704250] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.707495] vdf: detected capacity change from 0 to 4294967296 [ 3.713497] libphy: Fixed MDIO Bus: probed [ 3.719202] usbcore: registered new interface driver usbserial_generic [ 3.721435] usbserial: USB Serial support registered for generic [ 3.723547] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.728857] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.730556] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.732795] mousedev: PS/2 mouse device common for all mice [ 3.735720] rtc_cmos 00:05: RTC can wake from S4 [ 3.738249] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.739545] rtc_cmos 00:05: registered as rtc0 [ 3.744362] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.744863] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.747511] intel_pstate: CPU model not supported [ 3.752993] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.757845] hid: raw HID events driver (C) Jiri Kosina [ 3.760187] usbcore: registered new interface driver usbhid [ 3.762152] usbhid: USB HID core driver [ 3.763849] drop_monitor: Initializing network drop monitor service [ 3.766343] Initializing XFRM netlink socket [ 3.768367] NET: Registered protocol family 10 [ 3.772492] Segment Routing with IPv6 [ 3.774069] NET: Registered protocol family 17 [ 3.776096] mpls_gso: MPLS GSO support [ 3.781462] RAS: Correctable Errors collector initialized. [ 3.783502] AVX version of gcm_enc/dec engaged. [ 3.785141] AES CTR mode by8 optimization enabled [ 3.865791] sched_clock: Marking stable (3865747726, 0)->(4791888776, -926141050) [ 3.870215] registered taskstats version 1 [ 3.872774] Loading compiled-in X.509 certificates [ 3.875490] zswap: loaded using pool lzo/zbud [ 3.902303] Key type big_key registered [ 3.916189] Key type encrypted registered [ 3.918291] ima: No TPM chip found, activating TPM-bypass! [ 3.920317] ima: Allocated hash algorithm: sha1 [ 3.921995] ima: No architecture policies found [ 3.923822] evm: Initialising EVM extended attributes: [ 3.925813] evm: security.selinux [ 3.927181] evm: security.ima [ 3.928314] evm: security.capability [ 3.929583] evm: HMAC attrs: 0x1 [ 3.932870] rtc_cmos 00:05: setting system clock to 2025-11-17 01:48:37 UTC (1763344117) [ 3.942711] debug: unmapping init [mem 0xffffffff85803000-0xffffffff859fffff] [ 3.945771] debug: unmapping init [mem 0xffffffff84582000-0xffffffff84858fff] [ 3.954129] Write protecting the kernel read-only data: 28672k [ 3.957478] debug: unmapping init [mem 0xffffffff82c03000-0xffffffff82dfffff] [ 3.960265] debug: unmapping init [mem 0xffffffff83514000-0xffffffff835fffff] [ 4.000359] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.009423] systemd[1]: Detected virtualization kvm. [ 4.011296] systemd[1]: Detected architecture x86-64. [ 4.013461] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.036365] systemd[1]: No hostname configured. [ 4.038379] systemd[1]: Set hostname to . [ 4.040469] random: systemd: uninitialized urandom read (16 bytes read) [ 4.042825] systemd[1]: Initializing machine ID from random generator. [ 4.188730] random: systemd: uninitialized urandom read (16 bytes read) [ 4.191615] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.197076] random: systemd: uninitialized urandom read (16 bytes read) [ 4.198682] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.204782] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.799221] device-mapper: uevent: version 1.0.3 [ 4.801808] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK [0[ 5.403451] random: fast init done m] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.489371] virtio_net virtio0 ens2: renamed from eth0 [ 5.504479] scsi host0: ata_piix [ 5.606910] scsi host1: ata_piix [ 5.608668] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.610992] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.201966] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.435816] random: crng init done [ 10.437280] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.820533] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.960876] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.211335] SELinux: Disabled at runtime. [ 12.277241] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.286118] systemd[1]: Detected virtualization kvm. [ 12.288267] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.789083] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.792343] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.796860] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.802367] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.805662] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.813325] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.821233] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root.[ 12.886333] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.260722] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.525491] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.580730] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.662293] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.680145] EDAC sbridge: Ver: 1.1.2 [ 15.376206] Key type dns_resolver registered [ 15.674792] NFS: Registering the id_resolver key type [ 15.676616] Key type id_resolver registered [ 15.677794] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg427-server login: [ 65.810029] libcfs: loading out-of-tree module taints kernel. [ 65.842668] Key type ._llcrypt registered [ 65.844773] Key type .llcrypt registered [ 65.946934] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 80.876654] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 81.700383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 81.716847] alg: No test for adler32 (adler32-zlib) [ 83.027946] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 83.649323] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 85.360090] Key type lgssc registered [ 86.788160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.702041] hrtimer: interrupt took 9032043 ns [ 100.881324] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 101.662765] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.516958] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 114.656911] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 120.519278] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.087495] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 146.004231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 146.046457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 146.076620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.386134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 147.423906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 147.558967] Lustre: lustre-MDT0000: new disk, initializing [ 147.641315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 147.658055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 151.337846] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.578921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 164.660436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 164.709639] Lustre: 6512:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 164.749370] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 164.751897] Lustre: Skipped 1 previous similar message [ 164.821627] Lustre: lustre-MDT0001: new disk, initializing [ 164.876851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.934316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 164.951541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 168.340932] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.761571] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 180.161922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 180.215692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 180.375023] Lustre: lustre-OST0000: new disk, initializing [ 180.377576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 180.428826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.733205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 181.740293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 181.823926] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 184.760762] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.570141] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 195.622296] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 195.687315] Lustre: lustre-OST0001: new disk, initializing [ 195.690045] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 195.737911] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.031368] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.830543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 205.852093] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 205.904041] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 208.675234] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 214.402621] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 224.730530] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing check_logdir /tmp/testlogs/ [ 228.145615] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing yml_node [ 232.092154] Lustre: DEBUG MARKER: Client: 2.16.61.46 [ 234.615939] Lustre: DEBUG MARKER: MDS: 2.16.61.46 [ 236.859492] Lustre: DEBUG MARKER: OSS: 2.16.61.46 [ 238.037933] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 20:52:31 EST 2025 [ 254.387261] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 255.790336] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 272.352385] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 272.354310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 272.359823] Lustre: Skipped 1 previous similar message [ 272.360564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 275.315468] LustreError: 12216:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 275.390704] Lustre: server umount lustre-MDT0000 complete [ 278.134614] LustreError: 6504:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344391 with bad export cookie 9572471447408710566 [ 278.137790] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 278.151461] LustreError: 6504:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 282.591786] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 282.593576] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 282.594818] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 282.594824] Lustre: Skipped 3 previous similar messages [ 282.611422] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 282.633070] Lustre: Skipped 3 previous similar messages [ 283.619564] LustreError: 9166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 283.620497] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 283.630070] LustreError: 9166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 283.632921] Lustre: Skipped 2 previous similar messages [ 284.300961] LustreError: 12417:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 284.304760] LustreError: 12417:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 284.425218] Lustre: server umount lustre-MDT0001 complete [ 287.689095] LustreError: 12620:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 287.696330] LustreError: 12620:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 287.763790] Lustre: server umount lustre-OST0000 complete [ 290.819556] LustreError: 12820:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 290.826466] LustreError: 12820:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 290.909873] Lustre: server umount lustre-OST0001 complete [ 295.323846] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 302.840182] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 311.796923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 317.288371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 323.587639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 330.315941] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 337.574722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 337.646744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 337.965399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 338.018605] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 338.086806] Lustre: lustre-MDT0000: new disk, initializing [ 338.164597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 338.183153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 341.654597] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 351.438525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 351.502832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 351.572614] Lustre: 15004:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 351.626053] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 351.628230] Lustre: Skipped 1 previous similar message [ 351.732429] Lustre: lustre-MDT0001: new disk, initializing [ 351.843212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 351.877558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 351.892170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 355.784728] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 362.428781] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 365.469556] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 371.091820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 371.166371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 371.376083] Lustre: lustre-OST0000: new disk, initializing [ 371.383398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 371.467215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 373.319106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 373.327197] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 373.425225] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 376.177974] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 382.298556] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 385.411571] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 385.554600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 388.805708] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 388.982308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 394.209945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 394.213377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 394.218239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 394.248132] Lustre: Skipped 1 previous similar message [ 396.541184] LustreError: 17945:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 396.545679] LustreError: 17945:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 396.632594] Lustre: server umount lustre-OST0000 complete [ 399.331503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 399.336762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 399.347179] Lustre: Skipped 1 previous similar message [ 399.351635] Lustre: Skipped 1 previous similar message [ 405.390711] LustreError: 18146:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 405.393067] LustreError: 18146:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 405.509572] Lustre: server umount lustre-MDT0000 complete [ 408.707370] LustreError: 15788:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344522 with bad export cookie 9572471447408712106 [ 408.711693] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 408.720354] LustreError: 15788:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 408.937795] Lustre: server umount lustre-MDT0001 complete [ 412.727846] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 20:55:25 (1763344525) [ 414.957335] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 416.618606] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 20:55:29 (1763344529) [ 422.475236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.789401] LustreError: 19018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 422.861152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.474811] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 428.001793] LustreError: 19018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 431.874601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 435.199648] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 439.757758] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 442.846878] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 447.634281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 452.278918] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 457.027456] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 478.692188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 478.698077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 478.701626] LustreError: Skipped 1 previous similar message [ 478.714469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 478.721408] Lustre: Skipped 4 previous similar messages [ 483.895294] LustreError: 21285:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 483.899905] LustreError: 21285:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 483.977858] Lustre: server umount lustre-OST0000 complete [ 488.417944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 488.421679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.427253] Lustre: Skipped 1 previous similar message [ 492.836861] Lustre: server umount lustre-MDT0000 complete [ 494.054589] LustreError: 21178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.066685] LustreError: 21178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 496.310243] LustreError: 18997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344609 with bad export cookie 9572471447408712988 [ 496.315203] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 496.329209] LustreError: 18997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 496.698929] Lustre: server umount lustre-MDT0001 complete [ 505.797850] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 508.493384] Key type lgssc unregistered [ 508.942716] LNet: 22221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 508.971145] LNetError: 22221:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 508.997806] LNet: Removed LNI 192.168.204.127@tcp [ 509.893429] Key type .llcrypt unregistered [ 509.898619] Key type ._llcrypt unregistered [ 527.060556] Key type ._llcrypt registered [ 527.062212] Key type .llcrypt registered [ 527.209115] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 528.002912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 528.051987] alg: No test for adler32 (adler32-zlib) [ 529.067953] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 529.290582] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 530.999161] Key type lgssc registered [ 532.347504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 540.042244] loop: module loaded [ 540.114895] loop0: detected capacity change from 0 to 204800000 [ 540.281645] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 540.292022] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 540.520819] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 540.566714] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 545.517428] loop0: detected capacity change from 0 to 204800000 [ 545.700361] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 545.707856] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 545.715963] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 545.833547] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 550.243896] loop0: detected capacity change from 0 to 204800000 [ 554.739074] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 554.795956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 554.805792] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.006344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 556.073092] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 556.145209] Lustre: lustre-MDT0000: new disk, initializing [ 556.219521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.237546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 559.267627] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 568.010670] loop1: detected capacity change from 0 to 204800000 [ 573.573274] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 573.653413] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 573.996833] Lustre: lustre-OST07c6: new disk, initializing [ 574.007359] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 574.021721] Lustre: Skipped 1 previous similar message [ 574.122767] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 579.612674] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 581.723519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 581.745461] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 581.811052] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 585.287353] Lustre: 24361:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 585.302915] Lustre: 24361:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 585.310101] Lustre: 24361:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 585.313854] Lustre: 24361:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 585.316826] Lustre: 24361:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 585.319784] Lustre: 24361:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 599.899367] Lustre: 24360:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 599.905816] Lustre: 24360:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 599.910446] Lustre: 24360:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 599.914698] Lustre: 24360:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 599.918915] Lustre: 24360:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 599.922097] Lustre: 24360:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 604.443438] Lustre: 24360:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 604.447805] Lustre: 24360:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 604.450970] Lustre: 24360:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 604.486233] Lustre: 24360:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 604.502348] Lustre: 24360:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 604.516924] Lustre: 24360:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 604.524529] Lustre: 24360:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 604.535793] Lustre: 24360:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 604.543373] Lustre: 24360:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 604.555562] Lustre: 24360:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 604.568624] Lustre: 24360:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 604.587878] Lustre: 24360:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 607.759728] Lustre: 24359:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 607.764298] Lustre: 24359:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 607.777276] Lustre: 24359:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 607.784083] Lustre: 24359:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 607.791610] Lustre: 24359:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 607.797358] Lustre: 24359:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 607.801264] Lustre: 24359:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 607.806797] Lustre: 24359:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 607.811108] Lustre: 24359:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 607.814061] Lustre: 24359:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 607.817351] Lustre: 24359:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 607.821119] Lustre: 24359:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 612.304458] Lustre: 24360:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 612.317868] Lustre: 24360:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 612.323770] Lustre: 24360:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 612.337514] Lustre: 24360:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 612.345878] Lustre: 24360:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 612.349335] Lustre: 24360:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 612.355784] Lustre: 24360:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 612.365317] Lustre: 24360:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 612.375267] Lustre: 24360:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 612.384043] Lustre: 24360:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 612.390703] Lustre: 24360:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 612.395326] Lustre: 24360:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 623.010410] Lustre: 24726:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 623.014730] Lustre: 24726:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 5 previous similar messages [ 623.018445] Lustre: 24726:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 623.022018] Lustre: 24726:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 623.033614] Lustre: 24726:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 623.039054] Lustre: 24726:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 623.048037] Lustre: 24726:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 623.063359] Lustre: 24726:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 623.069855] Lustre: 24726:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 623.079345] Lustre: 24726:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 623.089703] Lustre: 24726:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 623.105048] Lustre: 24726:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 673.761438] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 673.765816] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 673.778395] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 675.581767] LustreError: 26548:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 675.707884] Lustre: server umount lustre-OST07c6 complete [ 683.934658] LustreError: 27051:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 683.942202] LustreError: 27051:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 684.152383] Lustre: server umount lustre-MDT0000 complete [ 699.721108] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 702.464535] Key type lgssc unregistered [ 702.765644] LNet: 27885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.777274] LNetError: 27885:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 702.813704] LNet: Removed LNI 192.168.204.127@tcp [ 703.640431] Key type .llcrypt unregistered [ 703.642414] Key type ._llcrypt unregistered [ 722.532740] Key type ._llcrypt registered [ 722.534539] Key type .llcrypt registered [ 722.697717] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 723.719624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 723.818426] alg: No test for adler32 (adler32-zlib) [ 724.988939] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 725.240292] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 726.935178] Key type lgssc registered [ 727.950714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 734.963989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 735.018817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.450426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 739.906826] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 745.954707] LustreError: 29136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 745.985216] LustreError: 29136:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 747.442063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 747.760879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 752.870270] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 759.106795] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 762.918883] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 769.701977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 770.149168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 775.400277] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 780.380763] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 781.440704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 794.081925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 794.089142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 794.122476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 797.155485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 797.158269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 797.160489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 797.197521] Lustre: Skipped 1 previous similar message [ 800.068147] LustreError: 31597:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 800.131705] Lustre: server umount lustre-OST0000 complete [ 804.328924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 804.337252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 804.352994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 807.395046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 807.395430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 809.280509] LustreError: 31799:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 809.283286] LustreError: 31799:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 809.406937] Lustre: server umount lustre-MDT0000 complete [ 812.466754] LustreError: 29117:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344926 with bad export cookie 805041015861885340 [ 812.474862] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 812.490627] LustreError: 29117:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 812.606566] LustreError: 32000:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 812.608617] LustreError: 32000:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 812.769630] Lustre: server umount lustre-MDT0001 complete [ 820.497351] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 822.257487] Key type lgssc unregistered [ 822.482902] LNet: 32532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.493073] LNetError: 32532:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 822.503835] LNet: Removed LNI 192.168.204.127@tcp [ 823.107257] Key type .llcrypt unregistered [ 823.109506] Key type ._llcrypt unregistered [ 837.542901] Key type ._llcrypt registered [ 837.544803] Key type .llcrypt registered [ 837.622901] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 838.289190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 838.301836] alg: No test for adler32 (adler32-zlib) [ 839.293307] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 839.504953] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 841.151316] Key type lgssc registered [ 842.150918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 849.051509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 849.075531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 850.395403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 853.387690] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 855.524482] LustreError: 33762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 855.542529] LustreError: 33762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 859.617491] LustreError: 33762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 860.532855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 860.808312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 864.415853] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 868.836760] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 872.119700] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 876.899943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 877.149306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 881.756924] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 887.753321] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 894.671804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 905.200699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 905.204225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 905.212101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 905.234426] Lustre: Skipped 1 previous similar message [ 910.310067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 910.312930] Lustre: Skipped 1 previous similar message [ 910.709481] LustreError: 36039:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 910.941499] Lustre: server umount lustre-OST0000 complete [ 915.426808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 915.432649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 920.547649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 920.550453] Lustre: Skipped 2 previous similar messages [ 920.910225] LustreError: 36240:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 920.913613] LustreError: 36240:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 920.994746] Lustre: server umount lustre-MDT0000 complete [ 924.277202] LustreError: 35381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345037 with bad export cookie 4880279958282328378 [ 924.278677] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 924.287903] LustreError: 35381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 924.416506] LustreError: 36442:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 924.423452] LustreError: 36442:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 924.608764] Lustre: server umount lustre-MDT0001 complete [ 934.337716] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 936.549651] Key type lgssc unregistered [ 936.766205] LNet: 36973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.773265] LNetError: 36973:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.833307] LNet: Removed LNI 192.168.204.127@tcp [ 938.472585] Key type .llcrypt unregistered [ 938.473889] Key type ._llcrypt unregistered [ 957.263353] Key type ._llcrypt registered [ 957.265565] Key type .llcrypt registered [ 957.374816] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 958.237687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 958.253262] alg: No test for adler32 (adler32-zlib) [ 959.356839] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 959.560215] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 961.199170] Key type lgssc registered [ 962.254771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 969.012631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 969.032076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 970.377899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 973.261266] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 979.941323] LustreError: 38351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 979.948948] LustreError: 38351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 981.089736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 981.596046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 984.914615] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 990.025540] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 992.703603] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 997.785714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 998.050867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1002.849119] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1007.894051] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1015.471979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1024.993175] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.27@tcp (stopping) [ 1026.022410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.025170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1026.035939] Lustre: Skipped 2 previous similar messages [ 1026.050197] Lustre: Skipped 2 previous similar messages [ 1030.102201] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.27@tcp (stopping) [ 1030.758743] LustreError: 40618:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1030.823964] Lustre: server umount lustre-MDT0000 complete [ 1031.136200] LustreError: 38351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1031.150424] LustreError: 38351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1034.373769] LustreError: 39969:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345147 with bad export cookie 4118947942959645665 [ 1034.386738] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1034.389907] LustreError: 39969:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1035.210501] LustreError: 38347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1035.240278] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.27@tcp (stopping) [ 1036.257528] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1039.264296] LustreError: 38347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.264932] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1039.281167] LustreError: 38347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1039.283910] Lustre: Skipped 1 previous similar message [ 1040.545576] LustreError: 40821:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1040.558796] LustreError: 40821:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1040.663437] Lustre: server umount lustre-MDT0001 complete [ 1060.671085] LustreError: 41044:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1060.676616] LustreError: 41044:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1060.791816] Lustre: server umount lustre-OST0000 complete [ 1073.158563] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1075.312971] Key type lgssc unregistered [ 1075.512320] LNet: 41882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.516633] LNetError: 41882:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1075.530574] LNet: Removed LNI 192.168.204.127@tcp [ 1076.328157] Key type .llcrypt unregistered [ 1076.330925] Key type ._llcrypt unregistered [ 1095.508131] Key type ._llcrypt registered [ 1095.509470] Key type .llcrypt registered [ 1095.602169] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1096.367127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1096.383479] alg: No test for adler32 (adler32-zlib) [ 1097.356765] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 1097.540074] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1099.191186] Key type lgssc registered [ 1100.081118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1105.994178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1106.008691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.346717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1110.118669] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1116.652211] LustreError: 43260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.679901] LustreError: 43260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1117.104671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1117.436780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1120.737994] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1126.237491] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1129.526416] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1135.428101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1135.746927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.918818] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1146.659046] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1152.180926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1157.601806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1157.605360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.623581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1157.635526] Lustre: Skipped 1 previous similar message [ 1158.642222] Lustre: lustre-OST0000: Not available for connect from 192.168.204.27@tcp (stopping) [ 1162.722525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1162.737816] Lustre: Skipped 1 previous similar message [ 1162.820260] LustreError: 45505:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1162.932509] Lustre: server umount lustre-OST0000 complete [ 1179.104719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1179.122135] LustreError: Skipped 1 previous similar message [ 1179.136373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.157142] Lustre: Skipped 1 previous similar message [ 1179.162070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1183.203520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1183.210386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1183.223334] Lustre: Skipped 1 previous similar message [ 1184.826405] LustreError: 45881:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1184.834352] LustreError: 45881:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1184.965448] Lustre: server umount lustre-MDT0000 complete [ 1187.777715] LustreError: 43239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345301 with bad export cookie 15083383633038623137 [ 1187.781124] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1187.787642] LustreError: 43239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1202.144043] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1202.222273] LustreError: 46083:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1202.227439] LustreError: 46083:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1202.367624] Lustre: server umount lustre-MDT0001 complete [ 1210.667696] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1213.045414] Key type lgssc unregistered [ 1213.360150] LNet: 46618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1213.370744] LNetError: 46618:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1213.382554] LNet: Removed LNI 192.168.204.127@tcp [ 1214.127859] Key type .llcrypt unregistered [ 1214.135479] Key type ._llcrypt unregistered [ 1237.582369] Key type ._llcrypt registered [ 1237.585028] Key type .llcrypt registered [ 1237.716237] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1238.881405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1238.889752] alg: No test for adler32 (adler32-zlib) [ 1239.967588] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 1240.182494] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1241.855221] Key type lgssc registered [ 1243.097764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1250.703732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1250.739311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1252.221940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1255.647862] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1261.543159] LustreError: 48015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.565551] LustreError: 48015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1263.342935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1263.723726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1267.353139] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1272.509392] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1275.672205] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1281.562334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1281.857981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1286.273704] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1290.755898] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1297.215503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1300.650537] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1303.580944] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1315.274235] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-11-1621h10m27s [ 1316.410858] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1317.861390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.866116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1317.874834] Lustre: Skipped 1 previous similar message [ 1317.886909] Lustre: Skipped 3 previous similar messages [ 1318.928955] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.27@tcp (stopping) [ 1323.018753] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.27@tcp (stopping) [ 1323.022268] Lustre: Skipped 3 previous similar messages [ 1325.030492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1325.033203] Lustre: Skipped 2 previous similar messages [ 1331.158647] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.27@tcp (stopping) [ 1331.161939] Lustre: Skipped 1 previous similar message [ 1332.198613] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1332.317418] LustreError: 50585:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1332.488676] Lustre: server umount lustre-MDT0000 complete [ 1335.630821] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1337.300450] LustreError: 50325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.271652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.422036] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1341.664610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1341.735115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1342.707888] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1344.707632] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1347.947880] Lustre: DEBUG MARKER: Wait for df (36068) ... [ 1349.562711] Lustre: DEBUG MARKER: done [ 1353.191672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.196943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1353.206595] Lustre: Skipped 2 previous similar messages [ 1353.221461] Lustre: Skipped 4 previous similar messages [ 1359.235071] LustreError: 51643:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1359.246520] LustreError: 51643:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1359.371413] Lustre: server umount lustre-OST0000 complete [ 1363.424984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1363.426800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1368.685745] LustreError: 51845:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1368.693686] LustreError: 51845:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1368.778563] Lustre: server umount lustre-MDT0000 complete [ 1369.578924] LustreError: 48728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1369.587091] LustreError: 48728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1372.488694] LustreError: 47997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345486 with bad export cookie 3528373199366995958 [ 1372.501069] LustreError: 47997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1372.501726] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1372.622119] LustreError: 52046:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1372.632911] LustreError: 52046:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1372.815937] Lustre: server umount lustre-MDT0001 complete [ 1381.646970] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1384.081822] Key type lgssc unregistered [ 1384.377775] LNet: 52578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1384.389477] LNetError: 52578:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1384.407805] LNet: Removed LNI 192.168.204.127@tcp [ 1385.115866] Key type .llcrypt unregistered [ 1385.118655] Key type ._llcrypt unregistered [ 1414.731687] Key type ._llcrypt registered [ 1414.733191] Key type .llcrypt registered [ 1414.819446] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 1427.814334] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1429.104872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1429.173884] alg: No test for adler32 (adler32-zlib) [ 1430.274490] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 1430.469205] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1432.127154] Key type lgssc registered [ 1433.280890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1441.169938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1446.820042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1451.917487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1456.845453] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1462.615678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1462.656524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1462.672603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1463.919067] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1463.986340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1464.084643] Lustre: lustre-MDT0000: new disk, initializing [ 1464.183488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1464.200979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1467.969538] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1476.849679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1476.937091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1477.029890] Lustre: 56526:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1477.074396] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1477.077438] Lustre: Skipped 1 previous similar message [ 1477.153687] Lustre: lustre-MDT0001: new disk, initializing [ 1477.234263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1477.261850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1477.273623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1480.680133] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1487.864382] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1491.468333] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1497.851523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1497.940854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1498.162153] Lustre: lustre-OST0000: new disk, initializing [ 1498.166369] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1498.269775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1503.730197] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1505.864136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1505.885055] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1506.018425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1511.378777] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1515.499844] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1515.681808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1519.450295] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1519.680095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1526.757323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.766507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1526.799231] Lustre: Skipped 1 previous similar message [ 1526.822222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1527.734412] LustreError: 59469:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1527.915337] Lustre: server umount lustre-OST0000 complete [ 1531.879991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.888828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1531.898806] Lustre: Skipped 1 previous similar message [ 1536.999296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1537.007670] Lustre: Skipped 1 previous similar message [ 1537.696535] LustreError: 59679:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1537.712215] LustreError: 59679:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1537.905280] Lustre: server umount lustre-MDT0000 complete [ 1541.509418] LustreError: 56520:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345655 with bad export cookie 1768979443653599643 [ 1541.518333] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1541.524689] LustreError: 56520:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1541.680817] LustreError: 59882:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1541.684381] LustreError: 59882:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1541.995269] Lustre: server umount lustre-MDT0001 complete [ 1550.352475] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 21:14:22 (1763345662) [ 1556.701703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1557.113928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1560.407350] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1562.608570] LustreError: 60552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1562.626369] LustreError: 60552:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1566.693748] LustreError: 60552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.409350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1568.690257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1572.475356] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1577.935143] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1582.346924] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1589.645193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1590.104465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1595.890842] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1601.461698] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1611.205663] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-11-1621h15m23s [ 1613.019132] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1619.311342] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1621.978973] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1622.276175] LustreError: 60549:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1642.505509] Lustre: lustre-MDT0000: Client cda936fe-9fff-48b7-891e-7b97f9e76dab (at 192.168.204.27@tcp) reconnecting [ 1662.375125] LustreError: 60549:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1664.698486] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1667.464904] Lustre: DEBUG MARKER: done [ 1674.211987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1674.219199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1674.224256] Lustre: Skipped 1 previous similar message [ 1674.251808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1675.076847] LustreError: 63563:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1675.104493] LustreError: 63563:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1675.203586] Lustre: server umount lustre-OST0000 complete [ 1679.328865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1679.345242] Lustre: Skipped 1 previous similar message [ 1679.350487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1679.353609] Lustre: Skipped 1 previous similar message [ 1685.157324] LustreError: 63765:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1685.163432] LustreError: 63765:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1685.393175] Lustre: server umount lustre-MDT0000 complete [ 1685.484070] LustreError: 63269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1685.503299] LustreError: 63269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1688.972709] LustreError: 60533:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345802 with bad export cookie 1768979443653600525 [ 1688.981305] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1688.993246] LustreError: 60533:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1689.364334] Lustre: server umount lustre-MDT0001 complete [ 1698.969612] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1701.671097] Key type lgssc unregistered [ 1702.000112] LNet: 64499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.016446] LNetError: 64499:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.036123] LNet: Removed LNI 192.168.204.127@tcp [ 1702.877332] Key type .llcrypt unregistered [ 1702.882255] Key type ._llcrypt unregistered [ 1734.494137] Key type ._llcrypt registered [ 1734.495659] Key type .llcrypt registered [ 1734.610595] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 1748.373976] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1749.123855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1749.176062] alg: No test for adler32 (adler32-zlib) [ 1750.128828] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 1750.257210] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1751.898199] Key type lgssc registered [ 1753.180364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1762.025836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1767.918391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1773.437351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1778.637476] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1783.956990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1784.020912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1784.035750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.224090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1785.249717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1785.307401] Lustre: lustre-MDT0000: new disk, initializing [ 1785.350539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1785.363653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1788.809691] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1797.158927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1797.217971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1797.284286] Lustre: 68448:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1797.328666] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1797.335702] Lustre: Skipped 1 previous similar message [ 1797.428521] Lustre: lustre-MDT0001: new disk, initializing [ 1797.517712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1797.566767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1797.574733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1800.528786] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1807.522969] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1810.891225] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1816.912300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1816.965953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1817.151990] Lustre: lustre-OST0000: new disk, initializing [ 1817.157965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1817.233838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1822.304242] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1825.377610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1825.410572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1825.495659] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1829.739695] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1833.527277] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1833.696892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1837.443684] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1837.655710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1841.122988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1841.124234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.132812] LustreError: Skipped 1 previous similar message [ 1841.141820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1841.145440] Lustre: Skipped 1 previous similar message [ 1844.195857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1844.200582] Lustre: Skipped 2 previous similar messages [ 1845.628117] LustreError: 71387:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1845.748930] Lustre: server umount lustre-OST0000 complete [ 1849.314115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.324106] Lustre: Skipped 1 previous similar message [ 1849.324291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1854.435263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1854.447558] Lustre: Skipped 2 previous similar messages [ 1855.210333] LustreError: 71589:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1855.216997] LustreError: 71589:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1855.278067] Lustre: server umount lustre-MDT0000 complete [ 1858.084828] LustreError: 68440:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345971 with bad export cookie 7765452755043419107 [ 1858.087083] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1858.089388] LustreError: 68440:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1858.203437] LustreError: 71790:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1858.210486] LustreError: 71790:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1858.391053] Lustre: server umount lustre-MDT0001 complete [ 1865.527747] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 21:19:38 (1763345978) [ 1866.754590] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1868.313962] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 21:19:41 (1763345981) [ 1879.529139] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1883.463409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1883.791762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.624942] Lustre: Failing over lustre-MDT0000 [ 1884.636607] LustreError: 72872:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1884.650589] LustreError: 72872:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1884.656462] LustreError: 72872:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1884.751074] LustreError: 72924:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1884.753515] LustreError: 72924:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1884.914499] Lustre: server umount lustre-MDT0000 complete [ 1890.681629] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:20:03 (1763346003) [ 1896.615366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1897.041595] LustreError: 73490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.062839] LustreError: 73490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1897.094903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.394987] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1902.563154] LustreError: 73490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1906.658379] LustreError: 73490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1907.414607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1907.757937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1910.994884] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1916.414449] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1919.967171] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1925.441719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1925.676579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1930.592154] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1935.543231] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1950.712741] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1956.647736] LustreError: 75859:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1956.654026] LustreError: 75859:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1956.771437] Lustre: server umount lustre-MDT0000 complete [ 1957.355881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.360211] LustreError: 75722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1957.363809] Lustre: Skipped 2 previous similar messages [ 1957.392967] LustreError: 75722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1959.968883] LustreError: 73472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346073 with bad export cookie 7765452755043420262 [ 1959.982837] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1960.232883] Lustre: server umount lustre-MDT0001 complete [ 1963.116898] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1967.177599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1975.021441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1978.847536] Lustre: 66660:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346076/real 1763346076] req@0000000089c9d696 x1849002151712128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763346092 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1978.892402] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.959299] Lustre: 66660:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346078/real 1763346078] req@0000000043f7846e x1849002151712256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763346094 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1983.967321] Lustre: 66657:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346081/real 1763346081] req@0000000063a9a87d x1849002151712384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763346097 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1985.067482] LustreError: 66656:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003285a630 x1849002151713280/t0(0) o250->MGC192.168.204.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1985.322767] LustreError: 75097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1985.398419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1985.498273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1986.401914] Lustre: 66658:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346083/real 1763346083] req@000000002328f639 x1849002151712640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763346099 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1988.687897] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1991.119793] Lustre: 66660:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346088/real 1763346088] req@00000000561da720 x1849002151712768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763346104 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1993.697649] LustreError: 76918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1993.714084] LustreError: 76918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1996.321041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1997.618502] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1999.893107] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2004.768311] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2008.025737] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2014.128359] LustreError: 78319:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2014.132820] LustreError: 78319:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2014.280504] Lustre: server umount lustre-MDT0000 complete [ 2017.250804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2017.266355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.289499] LustreError: 77639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2017.761633] LustreError: 76060:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346131 with bad export cookie 7765452755043422166 [ 2017.772139] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2017.777386] LustreError: 76060:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2018.187603] Lustre: server umount lustre-MDT0001 complete [ 2024.223338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2032.823627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.145028] Lustre: 66659:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346131/real 1763346131] req@00000000249e0b7d x1849002151736448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763346147 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2034.174631] Lustre: 66659:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2034.187764] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.813777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2044.789752] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2044.796459] Lustre: Skipped 1 previous similar message [ 2047.449200] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2050.023524] LustreError: 79376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2050.056681] LustreError: 79376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2055.313896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2055.620902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2055.624096] Lustre: Skipped 2 previous similar messages [ 2059.271825] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2060.786357] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2064.901873] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2068.680600] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2076.130447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2076.131735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2076.162188] Lustre: Skipped 1 previous similar message [ 2076.173924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2076.180621] Lustre: Skipped 1 previous similar message [ 2076.553427] LustreError: 80775:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2076.563811] LustreError: 80775:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2076.676347] Lustre: server umount lustre-MDT0000 complete [ 2081.038857] LustreError: 80777:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346194 with bad export cookie 7765452755043423538 [ 2081.045057] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2081.060650] LustreError: 80777:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2081.338245] Lustre: server umount lustre-MDT0001 complete [ 2086.896980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2089.987588] Lustre: DEBUG MARKER: files compared the same [ 2092.031792] Lustre: server umount lustre-OST0000 complete [ 2110.015709] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 2112.848317] Key type lgssc unregistered [ 2113.201611] LNet: 82410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.218205] LNetError: 82410:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2113.233777] LNet: Removed LNI 192.168.204.127@tcp [ 2114.022341] Key type .llcrypt unregistered [ 2114.023755] Key type ._llcrypt unregistered [ 2136.103785] Key type ._llcrypt registered [ 2136.105694] Key type .llcrypt registered [ 2136.224564] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2137.800068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2137.859077] alg: No test for adler32 (adler32-zlib) [ 2138.935883] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 2139.173362] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 2140.855159] Key type lgssc registered [ 2142.537884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2151.152941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2151.213169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2153.032740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2157.704745] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug +malloc all [ 2163.168148] LustreError: 83812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.190564] LustreError: 83812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2164.197074] LustreError: 83812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.833775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2166.087481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2169.481406] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug +malloc all [ 2174.270405] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2177.599469] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2184.193571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2184.424545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.892392] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug +malloc all [ 2194.988243] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2198.776654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2209.247955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2209.251695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2209.262833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2209.266725] Lustre: Skipped 1 previous similar message [ 2212.407868] LustreError: 86058:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2212.553383] Lustre: server umount lustre-OST0000 complete [ 2217.441461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2217.445085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.480359] Lustre: Skipped 1 previous similar message [ 2217.483232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2219.491171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2219.494676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2221.861696] LustreError: 86260:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2221.865609] LustreError: 86260:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2221.940952] Lustre: server umount lustre-MDT0000 complete [ 2224.609825] LustreError: 84876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.641568] LustreError: 84876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2225.112195] LustreError: 84530:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346338 with bad export cookie 396705641265986827 [ 2225.112685] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2225.122414] LustreError: 84530:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2225.314912] LustreError: 86461:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2225.321313] LustreError: 86461:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2225.492167] Lustre: server umount lustre-MDT0001 complete [ 2235.791964] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 2238.395497] Key type lgssc unregistered [ 2238.734248] LNet: 86993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2238.742655] LNetError: 86993:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.764914] LNet: Removed LNI 192.168.204.127@tcp [ 2239.607132] Key type .llcrypt unregistered [ 2239.615794] Key type ._llcrypt unregistered [ 2262.885863] Key type ._llcrypt registered [ 2262.887569] Key type .llcrypt registered [ 2262.947904] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2263.635560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2263.679283] alg: No test for adler32 (adler32-zlib) [ 2264.652124] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 2264.836668] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 2266.495167] Key type lgssc registered [ 2267.458683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2275.847726] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2275.868841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2291.618188] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2296.735538] LustreError: 88365:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.127@tcp: failed processing log, type 4: rc = -110 [ 2323.487166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.121372] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2334.459642] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2341.242843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.366694] Lustre: *** cfs_fail_loc=706, val=0*** [ 2341.757784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.815748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2344.941393] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2346.978382] LustreError: 88391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2346.989257] LustreError: 88391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2351.385559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2351.665569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2355.049732] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2359.711243] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2363.514688] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2366.945645] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2366.955455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.966705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2367.456453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2367.470663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2367.493291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2371.586488] LustreError: 90689:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2371.779354] Lustre: server umount lustre-OST0000 complete [ 2377.187787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2377.194134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.211074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2377.213072] Lustre: Skipped 1 previous similar message [ 2381.383561] LustreError: 90891:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2381.395588] LustreError: 90891:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2381.513690] Lustre: server umount lustre-MDT0000 complete [ 2383.334447] LustreError: 90428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.357831] LustreError: 90428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2384.497072] LustreError: 88372:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346498 with bad export cookie 17840512610737775505 [ 2384.503239] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2384.505199] LustreError: 88372:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2384.672370] LustreError: 91092:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2384.680153] LustreError: 91092:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2384.821119] Lustre: server umount lustre-MDT0001 complete [ 2393.528415] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 2396.345815] Key type lgssc unregistered [ 2396.648170] LNet: 91624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2396.664301] LNetError: 91624:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2396.684807] LNet: Removed LNI 192.168.204.127@tcp [ 2397.610526] Key type .llcrypt unregistered [ 2397.614943] Key type ._llcrypt unregistered [ 2421.623947] Key type ._llcrypt registered [ 2421.625550] Key type .llcrypt registered [ 2421.753788] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2423.101682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2423.112789] alg: No test for adler32 (adler32-zlib) [ 2424.197445] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 2424.465115] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 2426.191211] Key type lgssc registered [ 2427.539397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2434.233073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2434.251198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2439.386722] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2446.824890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2447.358543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2450.924962] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2452.450417] LustreError: 93693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.472149] LustreError: 93693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2456.547500] LustreError: 93693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.576596] LustreError: 93693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2459.003952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2459.314949] LustreError: 93693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2459.345993] LustreError: 93693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2459.364193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2464.424989] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2464.741743] LustreError: 93697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.760559] LustreError: 93697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2469.861471] LustreError: 93694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.870785] LustreError: 93694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2473.124350] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2473.139543] Lustre: MGS: Client da3ac302-4a32-4977-b486-824d8a7b54d9 (at 0@lo) reconnecting [ 2473.456197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2473.459081] LustreError: 93708:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2473.536972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2477.450441] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2490.337114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2490.345762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.375190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2493.920389] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2493.927159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2493.944840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2493.965886] Lustre: Skipped 1 previous similar message [ 2496.126208] LustreError: 95656:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2496.242911] Lustre: server umount lustre-OST0000 complete [ 2504.163463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2504.166518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.174130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2506.333944] LustreError: 95858:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2506.338331] LustreError: 95858:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2506.369136] Lustre: server umount lustre-MDT0000 complete [ 2509.281261] LustreError: 93697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.295599] LustreError: 93697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2509.728470] LustreError: 96060:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2509.731394] LustreError: 96060:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2509.874348] Lustre: server umount lustre-MDT0001 complete [ 2513.282858] LustreError: 96261:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2513.286514] LustreError: 96261:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2513.341820] Lustre: server umount MGS complete [ 2523.123118] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 21:30:35 (1763346635) [ 2553.028487] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 2563.430603] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2572.559308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2577.808178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2582.757637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2587.474863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2593.027900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2593.088532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2596.026330] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2603.909103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2603.961395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2604.167558] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2604.237553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2604.360307] Lustre: lustre-MDT0001: new disk, initializing [ 2604.434661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2604.456051] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2604.460716] Lustre: 100778:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2604.465126] LustreError: 100778:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2608.246665] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2617.582918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2617.633931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2617.854193] Lustre: lustre-OST0000: new disk, initializing [ 2617.863837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2617.930221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2623.283811] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2627.616510] Lustre: 101989:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000cc2ced43 x1849002859469952/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763346757 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2630.517465] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2636.849466] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2636.858569] Lustre: MGS: Client 4872a4a4-b8e8-4378-8961-142c115b2a65 (at 0@lo) reconnecting [ 2636.906229] Lustre: 100063:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2636.917373] LustreError: 100063:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2636.956990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2637.044214] Lustre: lustre-MDT0000: new disk, initializing [ 2637.219750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2637.260995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2637.379116] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2638.307629] Lustre: 102490:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000ab7298b1 x1849002859476224/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1763346767 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2640.912794] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2647.473206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2647.494369] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2647.621067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2672.611402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.617521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2672.619613] Lustre: Skipped 2 previous similar messages [ 2672.622579] Lustre: Skipped 1 previous similar message [ 2675.343927] LustreError: 103012:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2675.358545] LustreError: 103012:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2675.565251] Lustre: server umount lustre-OST0000 complete [ 2682.848529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2682.853019] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2682.858229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2682.861048] Lustre: Skipped 1 previous similar message [ 2682.864740] Lustre: Skipped 1 previous similar message [ 2685.247545] LustreError: 103213:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2685.255434] LustreError: 103213:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2685.287601] Lustre: server umount lustre-MDT0000 complete [ 2687.969415] LustreError: 100754:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2687.988261] LustreError: 100754:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2695.034434] Lustre: server umount lustre-MDT0001 complete [ 2699.024596] Lustre: server umount MGS complete [ 2710.384587] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 21:33:42 (1763346822) [ 2724.690133] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 2726.794729] Key type lgssc unregistered [ 2727.054665] LNet: 104954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2727.058973] LNetError: 104954:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2727.070608] LNet: Removed LNI 192.168.204.127@tcp [ 2727.780667] Key type .llcrypt unregistered [ 2727.782448] Key type ._llcrypt unregistered [ 2741.151973] Key type ._llcrypt registered [ 2741.156110] Key type .llcrypt registered [ 2741.258947] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2742.105330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2742.152161] alg: No test for adler32 (adler32-zlib) [ 2743.169257] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 2743.379836] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 2745.024634] Key type lgssc registered [ 2746.140897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2752.671329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2752.686108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2754.141442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2757.174516] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2764.215588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2764.258934] LustreError: 106140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2764.276605] LustreError: 106140:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2764.530436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2768.241734] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2773.655178] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2777.444335] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2783.765871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2784.069183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2790.292318] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2796.297339] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2798.510890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2808.804066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2808.811141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2808.816665] Lustre: Skipped 1 previous similar message [ 2808.817485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2813.449198] LustreError: 108385:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2813.617338] Lustre: server umount lustre-OST0000 complete [ 2819.044285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2819.064448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2819.068576] Lustre: Skipped 2 previous similar messages [ 2824.161400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2829.280703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2829.283984] Lustre: Skipped 2 previous similar messages [ 2831.327313] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2831.421775] LustreError: 108588:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2831.428479] LustreError: 108588:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2831.501672] Lustre: server umount lustre-MDT0000 complete [ 2834.408047] LustreError: 106134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2834.436509] LustreError: 106134:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2834.800372] LustreError: 106119:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346948 with bad export cookie 15629965585298065420 [ 2834.830227] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2834.932466] LustreError: 108792:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2834.941947] LustreError: 108792:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2835.213256] Lustre: server umount lustre-MDT0001 complete [ 2845.264086] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 2847.598894] Key type lgssc unregistered [ 2847.898467] LNet: 109324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2847.904951] LNetError: 109324:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2847.921467] LNet: Removed LNI 192.168.204.127@tcp [ 2848.609179] Key type .llcrypt unregistered [ 2848.610748] Key type ._llcrypt unregistered [ 2862.673341] Key type ._llcrypt registered [ 2862.674956] Key type .llcrypt registered [ 2862.803472] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2863.938997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2864.029896] alg: No test for adler32 (adler32-zlib) [ 2865.131181] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 2865.338378] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 2866.959178] Key type lgssc registered [ 2868.178252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2873.430908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2873.440086] LustreError: 110387:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2873.464880] Lustre: Skipped 1 previous similar message [ 2873.464902] LustreError: 110384:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2873.499429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2873.510070] LustreError: 110387:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 2873.566630] LustreError: 110387:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2873.588217] LustreError: 110387:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2875.054899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2882.736828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2883.006808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2886.587319] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2890.088483] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing lsmod [ 2892.112440] LustreError: 111563:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2892.121560] LustreError: 111562:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2892.135428] LustreError: 111563:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4980 [ 2892.139060] LustreError: 111563:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2892.145784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2892.147564] LustreError: 111563:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2892.524864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2894.798522] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.802203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2894.821575] Lustre: Skipped 2 previous similar messages [ 2894.828499] Lustre: Skipped 2 previous similar messages [ 2899.939961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2899.942208] Lustre: Skipped 2 previous similar messages [ 2905.059741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2905.068863] Lustre: Skipped 2 previous similar messages [ 2909.154855] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2909.237477] LustreError: 111750:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2909.359144] Lustre: server umount lustre-MDT0000 complete [ 2910.179306] LustreError: 110421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2910.203924] LustreError: 110421:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2912.773848] LustreError: 110398:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347026 with bad export cookie 6001681610886031861 [ 2912.788834] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2912.942444] LustreError: 111956:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2912.953278] LustreError: 111956:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2913.201821] Lustre: server umount lustre-MDT0001 complete [ 2926.468381] LustreError: 112157:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2926.471681] LustreError: 112157:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2926.579840] Lustre: server umount lustre-OST0000 complete [ 2933.027430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2933.332219] LustreError: 112636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.383213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2936.017327] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2938.856472] LustreError: 112636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.466869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2942.804490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2946.813141] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2952.241497] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2956.185781] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2961.903653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2962.107205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2967.555410] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2972.512154] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2978.487331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2983.913671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2983.915511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2983.927572] LustreError: Skipped 1 previous similar message [ 2983.952233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2984.759746] LustreError: 114883:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2984.762190] LustreError: 114883:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2984.892053] Lustre: server umount lustre-OST0000 complete [ 2989.029572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2989.045450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2989.078121] Lustre: Skipped 1 previous similar message [ 2989.084463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2989.089390] Lustre: Skipped 2 previous similar messages [ 2994.777988] LustreError: 115084:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2994.780690] LustreError: 115084:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2994.870065] Lustre: server umount lustre-MDT0000 complete [ 2998.067672] LustreError: 112615:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347111 with bad export cookie 6001681610886032547 [ 2998.071142] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2998.083782] LustreError: 112615:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2998.346867] Lustre: server umount lustre-MDT0001 complete [ 3007.716683] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 3011.192940] Key type lgssc unregistered [ 3011.526750] LNet: 115818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3011.545996] LNetError: 115818:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3011.572911] LNet: Removed LNI 192.168.204.127@tcp [ 3012.567583] Key type .llcrypt unregistered [ 3012.572809] Key type ._llcrypt unregistered [ 3035.458475] Key type ._llcrypt registered [ 3035.465159] Key type .llcrypt registered [ 3035.557328] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 3036.277407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3036.393474] alg: No test for adler32 (adler32-zlib) [ 3037.429756] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 3037.662153] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 3039.303163] Key type lgssc registered [ 3040.272750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3046.225190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3046.243633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3047.752081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3050.995414] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3058.147593] LustreError: 117214:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3058.159190] LustreError: 117214:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3058.497797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3058.744576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3061.751483] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3066.095446] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3069.005519] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3074.615638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3074.898470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.163095] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3085.263658] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3093.370120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3100.229884] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3102.548475] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3107.386149] LustreError: 119586:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3107.511851] Lustre: server umount lustre-OST0000 complete [ 3113.953229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3113.965566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3113.971067] Lustre: Skipped 1 previous similar message [ 3119.081893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3119.084656] Lustre: Skipped 1 previous similar message [ 3121.128983] LustreError: 119791:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3121.135200] LustreError: 119791:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3121.191588] Lustre: server umount lustre-MDT0000 complete [ 3124.200123] LustreError: 117215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3124.215620] LustreError: 117215:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3124.349991] LustreError: 119439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347237 with bad export cookie 473330156750598078 [ 3124.350546] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3124.355482] LustreError: 119439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3133.929985] LustreError: 119994:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3133.933013] LustreError: 119994:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3134.058418] Lustre: server umount lustre-MDT0001 complete [ 3141.653802] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 3143.551333] Key type lgssc unregistered [ 3143.750249] LNet: 120526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3143.754247] LNetError: 120526:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3143.771672] LNet: Removed LNI 192.168.204.127@tcp [ 3144.359908] Key type .llcrypt unregistered [ 3144.361683] Key type ._llcrypt unregistered [ 3158.792817] Key type ._llcrypt registered [ 3158.794464] Key type .llcrypt registered [ 3158.915487] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 3159.818593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3159.879131] alg: No test for adler32 (adler32-zlib) [ 3160.846784] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 3160.991735] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 3162.633880] Key type lgssc registered [ 3163.340297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3168.736490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3168.750633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3170.073762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3173.014958] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3178.515874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3178.745611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3181.269984] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3184.919623] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3187.495446] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3191.984837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3192.206689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.313673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3196.001480] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3199.614435] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3210.725496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3210.730442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3210.737764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3213.793554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3213.801271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3213.805062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3215.929139] LustreError: 123968:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3216.018847] Lustre: server umount lustre-OST0000 complete [ 3219.936223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3219.941301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.957645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3219.967216] Lustre: Skipped 1 previous similar message [ 3224.034805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3224.038550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.231377] LustreError: 124172:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3225.243584] LustreError: 124172:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3225.285495] Lustre: server umount lustre-MDT0000 complete [ 3227.975171] LustreError: 121705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347341 with bad export cookie 902257750195761796 [ 3227.982713] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3227.991441] LustreError: 121705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3228.147283] LustreError: 124374:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3228.150688] LustreError: 124374:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3228.264785] Lustre: server umount lustre-MDT0001 complete [ 3236.305147] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 3238.353957] Key type lgssc unregistered [ 3238.577066] LNet: 124906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3238.587126] LNetError: 124906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3238.602257] LNet: Removed LNI 192.168.204.127@tcp [ 3239.232171] Key type .llcrypt unregistered [ 3239.233605] Key type ._llcrypt unregistered [ 3257.784053] Key type ._llcrypt registered [ 3257.790533] Key type .llcrypt registered [ 3257.862981] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 3258.603233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3258.614264] alg: No test for adler32 (adler32-zlib) [ 3259.602547] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 3259.767953] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 3261.383454] Key type lgssc registered [ 3262.155854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3267.952976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3267.970925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.300366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3272.036828] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3278.571496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3278.917572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3281.937132] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3286.030276] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3288.693885] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3292.928173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3293.150177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3295.209526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3296.669828] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3300.164934] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3309.685609] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3311.850943] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3311.854195] Lustre: Skipped 2 previous similar messages [ 3314.384504] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3314.388216] Lustre: Skipped 2 previous similar messages [ 3315.220186] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3315.236954] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3323.613846] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.27@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3323.618524] Lustre: Skipped 2 previous similar messages [ 3329.030583] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.27@tcp 192.168.0.[2,10]@tcp [ 3332.360702] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.27/24@tcp in log lustre-MDT0000 [ 3332.365509] Lustre: Skipped 2 previous similar messages [ 3337.254371] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.27/24@tcp [ 3337.257549] Lustre: Skipped 1 previous similar message [ 3341.282101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3341.293835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3341.302729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3345.380226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3345.380943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3345.381678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.392679] Lustre: Skipped 1 previous similar message [ 3347.061358] LustreError: 129931:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3347.130299] Lustre: server umount lustre-OST0000 complete [ 3350.498306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3350.498382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3350.499849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3350.508385] Lustre: Skipped 1 previous similar message [ 3355.340057] LustreError: 130132:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3355.346457] LustreError: 130132:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3355.393076] Lustre: server umount lustre-MDT0000 complete [ 3355.616232] LustreError: 126283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3355.626492] LustreError: 126283:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3358.189447] LustreError: 126263:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347471 with bad export cookie 1231072571727349726 [ 3358.191914] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3358.195559] LustreError: 126263:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3358.316612] LustreError: 130333:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3358.320886] LustreError: 130333:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3358.464926] Lustre: server umount lustre-MDT0001 complete [ 3366.567588] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 3368.916613] Key type lgssc unregistered [ 3369.253917] LNet: 130865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3369.260676] LNetError: 130865:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3369.279813] LNet: Removed LNI 192.168.204.127@tcp [ 3370.181248] Key type .llcrypt unregistered [ 3370.183153] Key type ._llcrypt unregistered [ 3390.439710] Key type ._llcrypt registered [ 3390.441450] Key type .llcrypt registered [ 3390.512264] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 3391.104375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3391.131491] alg: No test for adler32 (adler32-zlib) [ 3392.105398] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 3392.241445] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 3393.847143] Key type lgssc registered [ 3394.480507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3399.814590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3399.834404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3401.169370] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3401.194591] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.27/24@tcp [ 3401.213721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3404.005821] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3410.475320] LustreError: 132387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3410.481401] LustreError: 132387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3410.512629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3410.766734] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3410.779925] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.27/24@tcp [ 3410.813625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3413.897837] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3418.163687] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3420.999890] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3427.637543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3428.217508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3433.173870] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3438.192382] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3445.624397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3456.995001] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3457.002067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3457.014552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3458.020816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3458.024393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3458.031803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3461.103460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3462.724101] LustreError: 134711:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3462.827498] Lustre: server umount lustre-OST0000 complete [ 3466.212282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3466.218383] Lustre: Skipped 1 previous similar message [ 3466.222515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3466.232901] Lustre: Skipped 2 previous similar messages [ 3471.328474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3471.332881] Lustre: Skipped 1 previous similar message [ 3472.050206] LustreError: 134912:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3472.057302] LustreError: 134912:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3472.158817] Lustre: server umount lustre-MDT0000 complete [ 3474.663120] LustreError: 134913:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347588 with bad export cookie 10727802689632052163 [ 3474.665454] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3474.668880] LustreError: 134913:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3474.837365] LustreError: 135114:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3474.843839] LustreError: 135114:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3474.984673] Lustre: server umount lustre-MDT0001 complete [ 3481.615170] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 3483.146505] Key type lgssc unregistered [ 3483.291466] LNet: 135646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3483.298937] LNetError: 135646:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3484.325589] LNet: Removed LNI 192.168.204.127@tcp [ 3484.732564] Key type .llcrypt unregistered [ 3484.734452] Key type ._llcrypt unregistered [ 3501.333418] Key type ._llcrypt registered [ 3501.344400] Key type .llcrypt registered [ 3501.433313] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 3512.927601] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 3513.619722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3513.682315] alg: No test for adler32 (adler32-zlib) [ 3514.750392] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 3514.963282] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 3516.591096] Key type lgssc registered [ 3517.444556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3524.110933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3528.573864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3533.323572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3538.618720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3543.062549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3543.095084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3543.109343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3544.277184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3544.301371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3544.358307] Lustre: lustre-MDT0000: new disk, initializing [ 3544.442859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3544.461123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3546.851211] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3553.656122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3553.717559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3553.788700] Lustre: 138996:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3553.841253] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3553.843342] Lustre: Skipped 1 previous similar message [ 3553.887567] Lustre: lustre-MDT0001: new disk, initializing [ 3553.970708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3554.010650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3554.018677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3556.950000] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3563.050444] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3565.718592] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3571.344262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3571.407549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3571.655787] Lustre: lustre-OST0000: new disk, initializing [ 3571.661542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3571.762881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3576.942725] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3578.414937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3578.422414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3578.503461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3583.113716] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3585.965962] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3586.153158] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3589.006304] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3589.196419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3593.696307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3593.701635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3593.710862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3594.214136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3594.214361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3594.231133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3596.546134] LustreError: 141934:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3596.618202] Lustre: server umount lustre-OST0000 complete [ 3599.332114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3599.340285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3599.340408] Lustre: Skipped 1 previous similar message [ 3599.346336] Lustre: Skipped 2 previous similar messages [ 3604.450282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3604.457633] Lustre: Skipped 1 previous similar message [ 3605.425701] LustreError: 142135:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3605.428379] LustreError: 142135:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3605.545888] Lustre: server umount lustre-MDT0000 complete [ 3608.624470] LustreError: 138990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347722 with bad export cookie 15343019131576184239 [ 3608.629563] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3608.635136] LustreError: 138990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3608.764692] LustreError: 142336:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3608.767693] LustreError: 142336:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3608.905825] Lustre: server umount lustre-MDT0001 complete [ 3615.050959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3615.453513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3618.364660] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3620.833342] LustreError: 142812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3620.843989] LustreError: 142812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3624.678829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3624.944692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3627.215771] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3630.929114] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3633.156824] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3637.292578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3637.504269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3641.768390] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3645.522860] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3647.796711] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3647.905623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3650.315112] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3650.465960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3656.557367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3656.592495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3656.690693] Lustre: lustre-OST0001: new disk, initializing [ 3656.693809] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3656.756091] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3660.384859] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3665.920405] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3667.008362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3667.020867] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3667.074605] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3668.179305] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3671.179407] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3671.344848] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3673.987583] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d2b91357000.ost_server_uuid 50 [ 3675.160557] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d2b91357000.ost_server_uuid in FULL state after 0 sec [ 3682.788705] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3682.789359] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3682.792279] LustreError: Skipped 1 previous similar message [ 3682.798139] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3682.804017] Lustre: Skipped 1 previous similar message [ 3686.590265] LustreError: 147073:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3686.592248] LustreError: 147073:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3686.640385] Lustre: server umount lustre-OST0001 complete [ 3687.916073] LustreError: 144409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3687.924082] LustreError: 144409:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3690.977160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3690.983185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3691.002919] Lustre: Skipped 1 previous similar message [ 3691.006898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3693.033193] LustreError: 144411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3693.047130] LustreError: 144411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3695.818618] LustreError: 147274:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3695.821706] LustreError: 147274:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3695.924173] Lustre: server umount lustre-OST0000 complete [ 3701.729033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3701.732908] LustreError: Skipped 1 previous similar message [ 3701.735391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3701.742302] Lustre: Skipped 1 previous similar message [ 3705.505298] LustreError: 147476:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3705.508420] LustreError: 147476:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3705.583773] Lustre: server umount lustre-MDT0000 complete [ 3708.321758] LustreError: 145816:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347821 with bad export cookie 15343019131576185121 [ 3708.326640] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3708.328267] LustreError: 145816:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3708.577282] Lustre: server umount lustre-MDT0001 complete [ 3722.427605] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 3724.576505] Key type lgssc unregistered [ 3724.779971] LNet: 148664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3724.791382] LNetError: 148664:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3724.810876] LNet: Removed LNI 192.168.204.127@tcp [ 3725.403866] Key type .llcrypt unregistered [ 3725.406041] Key type ._llcrypt unregistered [ 3748.850856] Key type ._llcrypt registered [ 3748.852635] Key type .llcrypt registered [ 3748.956917] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 3761.693993] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 3762.651302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3762.657924] alg: No test for adler32 (adler32-zlib) [ 3763.705924] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 3763.936482] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 3765.616823] Key type lgssc registered [ 3766.567728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3774.009190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3779.665182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3784.001718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3788.762785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3793.282761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3793.361763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3793.377578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3794.562437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3794.587695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3794.645770] Lustre: lustre-MDT0000: new disk, initializing [ 3794.726266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3794.740495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3797.912519] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3807.128904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3807.217739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3807.275141] Lustre: 152617:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3807.333783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3807.336565] Lustre: Skipped 1 previous similar message [ 3807.435379] Lustre: lustre-MDT0001: new disk, initializing [ 3807.544231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3807.579519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3807.590147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3811.013797] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3818.306462] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3822.057382] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3828.485421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3828.529303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3828.828104] Lustre: lustre-OST0000: new disk, initializing [ 3828.830410] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3828.925174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3834.132580] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3834.939051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3834.958088] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3835.063738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3840.801282] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3844.368826] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3844.504089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3847.546944] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3847.706625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3850.721042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3850.725249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.727817] LustreError: Skipped 1 previous similar message [ 3850.735085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3855.234619] LustreError: 155553:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3855.389543] Lustre: server umount lustre-OST0000 complete [ 3858.919538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3858.926497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3858.932929] Lustre: Skipped 1 previous similar message [ 3858.936467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3858.939928] Lustre: Skipped 1 previous similar message [ 3860.965181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.968263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3860.989697] Lustre: Skipped 1 previous similar message [ 3864.649794] LustreError: 155756:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3864.655550] LustreError: 155756:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3864.725410] Lustre: server umount lustre-MDT0000 complete [ 3867.737457] LustreError: 152607:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347981 with bad export cookie 82091050364005989 [ 3867.739089] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3867.746686] LustreError: 152607:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3867.856107] LustreError: 155957:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3867.859311] LustreError: 155957:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3868.029473] Lustre: server umount lustre-MDT0001 complete [ 3873.921167] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 21:53:06 (1763347986) [ 3885.220544] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 3894.536394] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 3904.294906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3911.140498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3916.357457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3921.325549] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3925.557140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3925.620363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3925.809389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3925.844587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3925.927664] Lustre: lustre-MDT0000: new disk, initializing [ 3925.996164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3926.013631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3928.679745] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3936.828937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3936.869223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3936.925549] Lustre: 158946:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3936.959474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3936.962356] Lustre: Skipped 1 previous similar message [ 3937.010594] Lustre: lustre-MDT0001: new disk, initializing [ 3937.084734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3937.106907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3937.114667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3939.587483] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3944.411767] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3946.713725] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3951.126897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3951.178423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3951.349478] Lustre: lustre-OST0000: new disk, initializing [ 3951.352665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3951.446894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3955.384735] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3959.343668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3959.356437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3959.439391] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3961.120479] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3964.495215] Lustre: Failing over lustre-OST0000 [ 3964.900772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3964.908742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3964.910720] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3966.584118] LustreError: 161394:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3966.590420] LustreError: 161394:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3966.674915] Lustre: server umount lustre-OST0000 complete [ 3982.835035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3983.080108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3983.089938] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3983.335816] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3985.135720] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 3985.136023] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3987.579231] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3990.792755] Lustre: Failing over lustre-MDT0000 [ 3990.898689] LustreError: 162379:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3990.901884] LustreError: 162379:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3990.981086] Lustre: server umount lustre-MDT0000 complete [ 3993.566151] LustreError: 158955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3993.569206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3993.588076] LustreError: 158955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3993.611232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3993.624694] Lustre: Skipped 1 previous similar message [ 3995.620922] LustreError: 158960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3995.636216] LustreError: 158960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3998.675133] LustreError: 158953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4000.741873] LustreError: 158955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4000.750380] LustreError: 158955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4005.860892] LustreError: 158960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4005.868556] LustreError: 158960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4007.305481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4007.370836] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4007.568630] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4008.915667] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4010.135711] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4013.054641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4013.057632] Lustre: Skipped 1 previous similar message [ 4013.075655] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4013.123546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4016.097897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4016.103314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4016.108844] Lustre: Skipped 2 previous similar messages [ 4016.113447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4016.117826] Lustre: Skipped 1 previous similar message [ 4021.890121] LustreError: 163263:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4021.893080] LustreError: 163263:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4021.987746] Lustre: server umount lustre-OST0000 complete [ 4028.387560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4028.388121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4028.394196] Lustre: Skipped 3 previous similar messages [ 4030.603994] Lustre: server umount lustre-MDT0000 complete [ 4033.129849] LustreError: 158940:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348146 with bad export cookie 82091050364008103 [ 4033.134476] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4033.141650] LustreError: 158940:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4033.320653] Lustre: server umount lustre-MDT0001 complete [ 4038.979374] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 4040.575056] Key type lgssc unregistered [ 4040.759305] LNet: 164196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4040.766384] LNetError: 164196:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4040.779850] LNet: Removed LNI 192.168.204.127@tcp [ 4041.271902] Key type .llcrypt unregistered [ 4041.274940] Key type ._llcrypt unregistered [ 4066.981618] Key type ._llcrypt registered [ 4066.983514] Key type .llcrypt registered [ 4067.058105] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4067.666624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4067.685684] alg: No test for adler32 (adler32-zlib) [ 4068.633252] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 4068.765081] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 4070.391176] Key type lgssc registered [ 4071.188676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4076.282485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4076.294171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4077.593891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4080.156761] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4085.774975] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4087.776910] LustreError: 165698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4087.793626] LustreError: 165698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4088.672676] LustreError: 165727:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4088.677924] LustreError: 165727:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4088.682514] LustreError: 165727:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 4088.762809] LustreError: 166249:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4088.869751] Lustre: server umount lustre-MDT0000 complete [ 4098.377613] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 4105.264845] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4111.596510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4116.343620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4120.048871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4123.727354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4127.468651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4127.517895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4127.668837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4127.692344] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4127.771699] Lustre: lustre-MDT0000: new disk, initializing [ 4127.847650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4127.857651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4130.256482] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4137.944649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4137.990890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4138.075139] Lustre: 168891:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4138.137969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4138.141706] Lustre: Skipped 1 previous similar message [ 4138.212622] Lustre: lustre-MDT0001: new disk, initializing [ 4138.302747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4138.334214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4138.341766] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4140.899576] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4145.886216] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4148.005962] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4152.497045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4152.559495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4152.745645] Lustre: lustre-OST0000: new disk, initializing [ 4152.750707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4152.819962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4157.137620] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4157.983866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4157.994213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4158.144477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4163.082352] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4166.561587] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4167.549545] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 21:58:00 (1763348280) [ 4363.231796] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.232113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4363.244099] Lustre: Skipped 2 previous similar messages [ 4363.250328] Lustre: Skipped 2 previous similar messages [ 4368.352712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4368.355792] Lustre: Skipped 2 previous similar messages [ 4373.472379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4373.477483] Lustre: Skipped 2 previous similar messages [ 4376.031169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4376.093897] LustreError: 172421:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4376.100239] LustreError: 172421:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4376.156542] Lustre: server umount lustre-MDT0000 complete [ 4378.594760] LustreError: 169769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.609655] LustreError: 169769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4378.712942] LustreError: 168886:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348492 with bad export cookie 7732700568743724667 [ 4378.713248] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4378.722056] LustreError: 168886:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4383.712515] LustreError: 170135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4383.713244] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.744971] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4384.931745] LustreError: 172628:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4384.934943] LustreError: 172628:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4385.035794] Lustre: server umount lustre-MDT0001 complete [ 4402.143189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4402.162557] LustreError: 172829:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4402.166398] LustreError: 172829:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4402.230556] Lustre: server umount lustre-OST0000 complete [ 4408.042453] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 4416.176368] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4423.735931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4428.770551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4433.716169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4438.197063] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4443.954487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4444.011519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4444.192033] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4444.222549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4444.286617] Lustre: lustre-MDT0000: new disk, initializing [ 4444.351726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4444.364420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4447.252665] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4454.807875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4454.848794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4454.910321] Lustre: 175170:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4454.969891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4454.973392] Lustre: Skipped 1 previous similar message [ 4455.036893] Lustre: lustre-MDT0001: new disk, initializing [ 4455.117699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4455.147362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4455.151850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4457.652638] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4462.766366] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4465.300333] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4469.591343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4469.680884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4469.842656] Lustre: lustre-OST0000: new disk, initializing [ 4469.845639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4473.273341] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4477.973440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4477.978154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4478.060223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4478.294228] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4480.645923] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4480.784663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4483.133494] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4483.265932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4488.162527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4488.162939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.169275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4488.173463] Lustre: Skipped 1 previous similar message [ 4488.179444] Lustre: Skipped 1 previous similar message [ 4490.683204] LustreError: 178105:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4490.697598] LustreError: 178105:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4490.767358] Lustre: server umount lustre-OST0000 complete [ 4496.353273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4496.360756] LustreError: Skipped 1 previous similar message [ 4496.363009] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4496.372347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4499.477966] LustreError: 178308:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4499.481145] LustreError: 178308:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4499.556856] Lustre: server umount lustre-MDT0000 complete [ 4501.822977] LustreError: 175161:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348615 with bad export cookie 7732700568743883497 [ 4501.823862] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4501.828720] LustreError: 175161:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4501.998587] Lustre: server umount lustre-MDT0001 complete [ 4507.169609] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:03:40 (1763348620) [ 4519.427618] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 4527.281482] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4535.462514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4539.966346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4544.271214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4549.048094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4553.762202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4553.811943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4553.961911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4553.987642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4554.043601] Lustre: lustre-MDT0000: new disk, initializing [ 4554.112470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4554.120961] Lustre: Skipped 1 previous similar message [ 4554.133654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4556.702971] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4564.755635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4564.815806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4564.874655] Lustre: 181499:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4565.064892] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4568.118604] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4574.319974] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4577.320559] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4582.353858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4582.400478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4586.693313] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4590.118056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4590.122341] Lustre: Skipped 1 previous similar message [ 4590.125227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4590.208124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4592.517821] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4600.291149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4600.292528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4600.299100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4600.301153] Lustre: Skipped 2 previous similar messages [ 4600.305336] Lustre: Skipped 2 previous similar messages [ 4603.252092] LustreError: 184044:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4603.265258] LustreError: 184044:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4603.336528] Lustre: server umount lustre-OST0000 complete [ 4605.933340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4610.531036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4610.536491] Lustre: Skipped 1 previous similar message [ 4611.673341] Lustre: server umount lustre-MDT0000 complete [ 4615.184232] LustreError: 182283:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348728 with bad export cookie 7732700568743884372 [ 4615.191979] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4615.198394] LustreError: 182283:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4615.596557] Lustre: server umount lustre-MDT0001 complete [ 4622.442435] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:05:35 (1763348735) [ 4636.247882] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 4643.920658] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4651.731163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4657.759464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4663.143861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4667.394519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4672.191787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4672.242666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4672.393904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4672.426819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4672.430525] Lustre: Skipped 3 previous similar messages [ 4672.493594] Lustre: lustre-MDT0000: new disk, initializing [ 4672.497498] Lustre: Skipped 2 previous similar messages [ 4672.569646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4672.572969] Lustre: Skipped 2 previous similar messages [ 4672.585036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4675.340896] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4682.834145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4682.912949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4683.003955] Lustre: 187436:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4683.218183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4686.148947] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4692.786548] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4696.444823] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4701.478042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4701.523596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4706.174600] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4710.579243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4712.726892] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4719.012727] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4719.020225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4719.027239] Lustre: Skipped 1 previous similar message [ 4719.036816] Lustre: Skipped 4 previous similar messages [ 4724.497821] LustreError: 189984:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4724.500671] LustreError: 189984:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4724.709618] Lustre: server umount lustre-OST0000 complete [ 4729.317300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4734.234421] Lustre: server umount lustre-MDT0000 complete [ 4734.433518] LustreError: 187448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4734.452360] LustreError: 187448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4737.508902] LustreError: 187430:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348851 with bad export cookie 7732700568743885464 [ 4737.512863] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4737.532995] LustreError: 187430:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4737.761810] Lustre: server umount lustre-MDT0001 complete [ 4745.487060] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 4747.466464] Key type lgssc unregistered [ 4747.668026] LNet: 190919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4747.675505] LNetError: 190919:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4747.698416] LNet: Removed LNI 192.168.204.127@tcp [ 4748.143059] Key type .llcrypt unregistered [ 4748.144743] Key type ._llcrypt unregistered [ 4766.648555] Key type ._llcrypt registered [ 4766.652936] Key type .llcrypt registered [ 4766.736600] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4767.597215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4767.628493] alg: No test for adler32 (adler32-zlib) [ 4768.636432] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 4768.838952] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 4770.543625] Key type lgssc registered [ 4771.668666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4778.698407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4778.719178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4780.227598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4783.885075] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4789.734558] LustreError: 192295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4789.757259] LustreError: 192295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4792.608459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4793.044494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4797.098202] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4803.031118] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4806.783530] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4813.602626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4814.061665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4819.674567] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4824.779190] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4825.339556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4844.009975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4844.021947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4844.033857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4846.050891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4846.059347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4846.069104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4847.418519] LustreError: 194567:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4847.578826] Lustre: server umount lustre-OST0000 complete [ 4851.176137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4851.192971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4851.198939] Lustre: Skipped 2 previous similar messages [ 4856.288080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4856.290844] Lustre: Skipped 1 previous similar message [ 4856.804120] LustreError: 194768:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4856.818805] LustreError: 194768:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4856.988505] Lustre: server umount lustre-MDT0000 complete [ 4859.872857] LustreError: 192277:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348973 with bad export cookie 17640178407659905787 [ 4859.888341] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4860.010761] LustreError: 194970:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4860.013318] LustreError: 194970:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4860.157113] Lustre: server umount lustre-MDT0001 complete [ 4869.519741] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 4871.494805] Key type lgssc unregistered [ 4871.764236] LNet: 195503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4871.775203] LNetError: 195503:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4871.792507] LNet: Removed LNI 192.168.204.127@tcp [ 4872.531399] Key type .llcrypt unregistered [ 4872.533089] Key type ._llcrypt unregistered [ 4892.557757] Key type ._llcrypt registered [ 4892.559578] Key type .llcrypt registered [ 4892.646426] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4893.736660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4893.763323] alg: No test for adler32 (adler32-zlib) [ 4894.765414] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 4895.032983] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 4896.687303] Key type lgssc registered [ 4897.764427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4904.800973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4904.815857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4906.201964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4909.195625] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4915.694853] LustreError: 196879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4915.729959] LustreError: 196879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4918.128982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4918.639735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4922.573388] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4928.893916] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4932.786388] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4938.540310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4938.978402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4943.087828] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4946.995952] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4951.275119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4958.689769] Lustre: lustre-OST0000: Not available for connect from 192.168.204.27@tcp (stopping) [ 4959.714830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4959.719498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4959.730434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4961.761816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4961.763062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4961.765311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4963.792727] Lustre: lustre-OST0000: Not available for connect from 192.168.204.27@tcp (stopping) [ 4963.804049] Lustre: Skipped 1 previous similar message [ 4968.908091] Lustre: lustre-OST0000: Not available for connect from 192.168.204.27@tcp (stopping) [ 4968.910520] Lustre: Skipped 2 previous similar messages [ 4970.975211] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4971.023518] LustreError: 199126:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4971.130022] Lustre: server umount lustre-OST0000 complete [ 4976.333685] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9d2b868f2800.ost_server_uuid 50 [ 4978.169273] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4995.552448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4995.565685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4995.583959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4997.956531] LustreError: 199495:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4997.963488] LustreError: 199495:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4998.224984] Lustre: server umount lustre-MDT0000 complete [ 5002.354473] LustreError: 196860:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349115 with bad export cookie 17649743211974609921 [ 5002.359955] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5002.570789] LustreError: 199697:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5002.577927] LustreError: 199697:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5002.774500] Lustre: server umount lustre-MDT0001 complete [ 5012.185286] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 22:12:04 (1763349124) [ 5018.738773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5019.122027] LustreError: 200370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5019.207967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5022.191378] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5024.225704] LustreError: 200370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5029.315509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5029.346536] LustreError: 200370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5029.669247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5033.317971] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5039.377903] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5042.943287] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5049.404261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5049.777364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5055.570448] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5060.241978] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5064.111217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5066.072800] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5066.131896] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5066.254633] Lustre: lustre-OST0001: new disk, initializing [ 5066.266253] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5066.358324] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5070.674253] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5072.509719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5072.515605] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5072.636637] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5076.479680] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5079.521550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5079.524980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5079.540172] Lustre: Skipped 1 previous similar message [ 5079.544417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5079.547326] Lustre: Skipped 4 previous similar messages [ 5084.788343] LustreError: 203575:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5084.793881] LustreError: 203575:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5084.827545] Lustre: server umount lustre-OST0000 complete [ 5087.184190] LustreError: 201966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5088.505416] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5088.625419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5091.331113] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5091.521475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5097.422106] LustreError: 201967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5097.440496] LustreError: 201967:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5108.707884] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5108.712886] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5108.723243] LustreError: Skipped 2 previous similar messages [ 5108.736089] Lustre: Skipped 2 previous similar messages [ 5111.613961] LustreError: 204298:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5111.622414] LustreError: 204298:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5111.868499] Lustre: server umount lustre-OST0001 complete [ 5116.896587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5116.900045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5116.928793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5116.931787] Lustre: Skipped 6 previous similar messages [ 5130.207174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5130.323387] LustreError: 204500:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5130.326537] LustreError: 204500:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5130.423302] Lustre: server umount lustre-MDT0000 complete [ 5133.762060] LustreError: 200349:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349247 with bad export cookie 17649743211974610992 [ 5133.764555] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5133.769423] LustreError: 200349:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5134.076821] Lustre: server umount lustre-MDT0001 complete [ 5139.402868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5142.918873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5146.225206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5149.545752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5155.750986] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 22:14:28 (1763349268) [ 5161.443760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5161.661244] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5161.711940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5162.017336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5165.968305] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5173.772617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5173.838821] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5173.869188] Lustre: 206200:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5177.439548] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5182.499626] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5185.997390] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5190.713473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5190.839641] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5195.077907] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5198.769354] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5200.460175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5203.783706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5203.882239] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5208.590353] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5209.277044] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5213.433755] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5219.298125] Lustre: lustre-OST0000: Not available for connect from 192.168.204.27@tcp (stopping) [ 5219.301448] Lustre: Skipped 6 previous similar messages [ 5219.815701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5219.823225] Lustre: Skipped 2 previous similar messages [ 5230.047567] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5230.093197] LustreError: 209397:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5230.101498] LustreError: 209397:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5230.152612] Lustre: server umount lustre-OST0000 complete [ 5234.640970] LustreError: 207825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5234.660867] LustreError: 207825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5245.409522] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5245.414074] LustreError: Skipped 1 previous similar message [ 5249.437754] Lustre: server umount lustre-OST0001 complete [ 5255.650791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.664029] Lustre: Skipped 2 previous similar messages [ 5266.400251] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5266.472617] LustreError: 209825:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5266.476084] LustreError: 209825:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5266.563124] Lustre: server umount lustre-MDT0000 complete [ 5269.375874] LustreError: 206191:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349382 with bad export cookie 17649743211974612392 [ 5269.377808] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5269.382632] LustreError: 206191:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5269.592502] Lustre: server umount lustre-MDT0001 complete [ 5273.785453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5276.901535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5279.845154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5282.484486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5288.364822] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 22:16:41 (1763349401) [ 5301.263912] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 5309.647161] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5319.216477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5325.095245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5330.938213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5337.886927] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5344.955893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5345.018974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5345.213294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5345.282371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5345.335464] Lustre: lustre-MDT0000: new disk, initializing [ 5345.415306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5345.417956] Lustre: Skipped 3 previous similar messages [ 5345.442762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5349.048495] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5357.300327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5357.367413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5357.430125] Lustre: 213852:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5357.501964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5357.505407] Lustre: Skipped 1 previous similar message [ 5357.585723] Lustre: lustre-MDT0001: new disk, initializing [ 5357.770851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5357.777692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5361.538242] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5368.400088] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5371.514378] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5378.028035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5378.104054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5378.327131] Lustre: lustre-OST0000: new disk, initializing [ 5378.335219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5383.557314] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5386.344121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5386.360103] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5386.532454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5390.356202] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5394.407329] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5394.572526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5398.395657] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5398.583195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5402.082498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5402.086087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5402.087911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5402.087918] Lustre: Skipped 16 previous similar messages [ 5402.094171] Lustre: Skipped 2 previous similar messages [ 5402.098316] LustreError: Skipped 1 previous similar message [ 5414.367380] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5414.411994] LustreError: 216792:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5414.416591] LustreError: 216792:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5414.542654] Lustre: server umount lustre-OST0000 complete [ 5423.410559] Lustre: server umount lustre-MDT0000 complete [ 5426.656113] LustreError: 213844:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349540 with bad export cookie 17649743211974614184 [ 5426.665808] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5426.671683] LustreError: 213844:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5427.055602] Lustre: server umount lustre-MDT0001 complete [ 5433.995591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5434.316523] LustreError: 217674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5434.338853] LustreError: 217674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5437.180310] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5443.014618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5446.104687] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5449.695210] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5452.337462] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5458.075355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5462.367070] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5466.649863] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5470.002277] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5470.146198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5473.019662] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5473.185456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5474.787242] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5480.674478] Lustre: server umount lustre-OST0000 complete [ 5485.564351] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5488.537836] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5514.965747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5515.314278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5515.322325] Lustre: Skipped 5 previous similar messages [ 5518.374301] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.383210] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5519.827443] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5525.232339] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5528.612389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.623077] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5536.791858] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 9 sec [ 5543.904941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5543.927571] Lustre: Skipped 4 previous similar messages [ 5545.984128] LustreError: 222082:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5545.987016] LustreError: 222082:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 5546.115953] Lustre: server umount lustre-OST0000 complete [ 5555.731030] Lustre: server umount lustre-MDT0000 complete [ 5559.243698] LustreError: 218392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349672 with bad export cookie 17649743211974615059 [ 5559.252836] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5559.616042] Lustre: server umount lustre-MDT0001 complete [ 5567.648992] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 22:21:20 (1763349680) [ 5572.972208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5573.272304] LustreError: 223156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5573.282437] LustreError: 223156:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5576.682737] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5583.900488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5586.835931] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5590.881185] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5593.776624] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5599.571386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5604.467413] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5610.140872] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5614.087659] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5619.613865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 5622.859486] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5623.020090] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5628.103717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5628.166452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5628.247865] Lustre: lustre-OST0001: new disk, initializing [ 5628.251046] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5633.051795] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5634.182554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5634.206461] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5634.412835] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5640.687189] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5644.445570] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5644.643131] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5648.589240] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5648.796491] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5655.016589] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5655.019974] LustreError: Skipped 4 previous similar messages [ 5660.134959] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5660.142587] Lustre: Skipped 20 previous similar messages [ 5665.247222] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5665.314740] Lustre: server umount lustre-OST0001 complete [ 5670.499739] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5670.823237] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5675.779803] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5676.032720] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5702.607286] LustreError: 224754:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5702.625052] LustreError: 224754:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 5704.415761] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5708.999762] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5709.811809] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5709.820534] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5709.821997] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5712.901791] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5714.353648] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5722.245128] Lustre: server umount lustre-OST0001 complete [ 5743.552725] Lustre: server umount lustre-OST0000 complete [ 5753.278563] Lustre: server umount lustre-MDT0000 complete [ 5756.308112] LustreError: 223135:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349869 with bad export cookie 17649743211974616032 [ 5756.310490] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5756.316653] LustreError: 223135:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5762.785037] Lustre: server umount lustre-MDT0001 complete [ 5768.236831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5771.804882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5775.071421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5778.457969] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5785.272727] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 22:24:57 (1763349897) [ 5791.530470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5791.735985] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5791.761920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5791.923674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5791.926866] Lustre: Skipped 5 previous similar messages [ 5796.104308] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5804.962814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5805.021216] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5805.046984] Lustre: 231176:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5808.513474] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5813.696456] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5817.529392] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5823.785854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5823.935977] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5829.250159] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5834.407533] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5843.554750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5843.668886] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5849.723882] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5854.786596] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5856.272974] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5859.797447] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5860.024130] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5863.653851] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5863.860283] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5867.361580] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d2ba02fa000.ost_server_uuid 50 [ 5868.812868] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d2ba02fa000.ost_server_uuid in FULL state after 0 sec [ 5869.727808] Lustre: Permanently deactivating lustre-OST0001 [ 5869.742409] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5872.173738] Lustre: Permanently reactivating lustre-OST0001 [ 5877.731734] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5877.738534] Lustre: Skipped 11 previous similar messages [ 5880.764084] LustreError: 235162:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5880.770589] LustreError: 235162:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 5880.811316] Lustre: server umount lustre-OST0001 complete [ 5890.272806] Lustre: server umount lustre-OST0000 complete [ 5899.608198] Lustre: server umount lustre-MDT0000 complete [ 5902.642624] LustreError: 231166:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350016 with bad export cookie 17649743211974617320 [ 5902.644367] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5902.672309] LustreError: 231166:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5903.021213] Lustre: server umount lustre-MDT0001 complete [ 5908.161046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5911.698862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5914.487754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5917.276026] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5922.903187] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 22:27:15 (1763350035) [ 5923.762421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5928.751475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5929.035596] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5929.057957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5929.065441] Lustre: Skipped 5 previous similar messages [ 5931.975928] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5938.776386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5938.877480] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5938.910771] Lustre: 237321:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5942.062749] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5946.312971] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5949.595934] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5954.862691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5954.939158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5955.112709] Lustre: Permanently deactivating lustre-OST0000 [ 5955.123141] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5961.075165] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5964.864231] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5966.341958] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5972.350683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5972.396748] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5972.399072] Lustre: Skipped 1 previous similar message [ 5977.331540] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5978.353996] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5982.650578] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5984.416038] Lustre: Permanently reactivating lustre-OST0000 [ 5984.427995] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5984.431031] Lustre: Skipped 2 previous similar messages [ 5991.446061] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5991.449304] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.468431] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5991.470971] Lustre: Skipped 1 previous similar message [ 5991.505830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.521041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5997.033256] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 5997.036870] LustreError: Skipped 6 previous similar messages [ 6001.097129] Lustre: server umount lustre-OST0001 complete [ 6004.196291] LustreError: 238947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6004.231721] LustreError: 238947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 6010.328336] Lustre: server umount lustre-OST0000 complete [ 6028.255313] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6028.409615] Lustre: server umount lustre-MDT0000 complete [ 6031.700702] LustreError: 237315:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350145 with bad export cookie 17649743211974618972 [ 6031.706051] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6031.708639] LustreError: 237315:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6032.052320] Lustre: server umount lustre-MDT0001 complete [ 6041.236668] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 6043.912321] Key type lgssc unregistered [ 6044.246466] LNet: 241718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6044.250770] LNetError: 241718:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6044.263588] LNet: Removed LNI 192.168.204.127@tcp [ 6044.933995] Key type .llcrypt unregistered [ 6044.935835] Key type ._llcrypt unregistered [ 6063.782269] Key type ._llcrypt registered [ 6063.783956] Key type .llcrypt registered [ 6063.886355] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 6064.575784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6064.792971] alg: No test for adler32 (adler32-zlib) [ 6065.883955] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 6066.073841] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 6067.707083] Key type lgssc registered [ 6068.562762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6071.976386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6078.794623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6078.815359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6080.255128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6084.207181] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6089.700175] LustreError: 243163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6089.712435] LustreError: 243163:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6092.792399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6092.848124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6092.907186] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6092.938160] Lustre: 243149:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6092.945398] Lustre: Permanently deactivating lustre-MDT0001 [ 6092.954192] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6093.254075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6096.917591] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6098.959333] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6102.048772] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6105.567455] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6111.720975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6112.083769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6117.424572] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6118.248245] LustreError: 244785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6123.035953] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6123.491426] LustreError: 244783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6124.532676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6129.642234] LustreError: 244784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6129.700260] LustreError: 244784:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6130.796325] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6131.298888] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6136.308599] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6136.319514] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6137.851443] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6142.970974] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6145.954456] Lustre: Permanently reactivating lustre-MDT0001 [ 6145.966409] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6145.970652] Lustre: Skipped 2 previous similar messages [ 6151.156780] Lustre: lustre-MDT0001: Client ae9d2dc6-3929-4b06-921d-87b79179746a (at 192.168.204.27@tcp) reconnecting [ 6154.839056] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6154.847249] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6154.851839] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6154.857295] LustreError: 246357:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6154.857665] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6154.863458] LustreError: 246357:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6154.873343] LustreError: 246357:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.127@tcp: cfg command failed: rc = -108 [ 6154.878275] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6154.878275] [ 6154.881529] LustreError: 243154:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6155.099367] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6157.379655] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6162.515894] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6162.705707] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6165.741672] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6165.921979] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6167.989584] Lustre: Permanently deactivating lustre-MDT0001 [ 6168.016887] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6168.020543] Lustre: Skipped 2 previous similar messages [ 6177.936283] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6184.586831] LustreError: 247235:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6185.443920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6185.447486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6185.475529] LustreError: 243162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6186.657425] LustreError: 247235:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 6186.662796] LustreError: 247235:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6186.747272] Lustre: server umount lustre-MDT0000 complete [ 6187.494331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6187.507098] Lustre: Skipped 2 previous similar messages [ 6190.694938] LustreError: 246275:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350304 with bad export cookie 900433572205045918 [ 6190.702400] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6190.718299] LustreError: 246275:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6190.920823] LustreError: 247436:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6190.940129] LustreError: 247436:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6191.170941] Lustre: server umount lustre-MDT0001 complete [ 6197.175169] LustreError: 247637:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6197.178242] LustreError: 247637:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6197.299593] Lustre: server umount lustre-OST0000 complete [ 6207.944866] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 22:32:00 (1763350320) [ 6217.055846] LustreError: 248643:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6217.061193] LustreError: 248643:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6217.230604] Lustre: server umount lustre-OST0001 complete [ 6222.390107] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 6231.090203] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 6240.450471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6246.965419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6253.501126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6259.971632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6266.981171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6267.049480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6267.272303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6267.283162] Lustre: Skipped 2 previous similar messages [ 6267.319613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6267.381865] Lustre: lustre-MDT0000: new disk, initializing [ 6267.481602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6267.499421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6271.152970] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6280.537262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6280.601393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6280.682464] Lustre: 250830:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6280.763298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6280.771144] Lustre: Skipped 1 previous similar message [ 6280.845800] Lustre: lustre-MDT0001: new disk, initializing [ 6280.977289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6281.020140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6281.043558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6284.541681] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6291.610372] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6295.135668] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6301.517804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6301.584783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6301.874740] Lustre: lustre-OST0000: new disk, initializing [ 6301.877564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6301.965898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6306.999414] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6309.461123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6309.476190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6309.544203] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6314.566910] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6316.657455] LustreError: 250838:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6324.545510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6324.647900] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6324.769075] Lustre: lustre-OST0001: new disk, initializing [ 6324.778630] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6326.755213] LustreError: 250838:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6326.791833] LustreError: 250838:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6329.833960] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6333.000335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6333.010767] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6333.149398] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6336.427633] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6336.866819] LustreError: 250838:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6337.823157] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6342.647767] Lustre: lustre-OST0001: Not available for connect from 192.168.204.27@tcp (stopping) [ 6343.654555] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6343.658722] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6343.683663] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6343.689453] Lustre: Skipped 1 previous similar message [ 6345.543314] LustreError: 254321:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6345.545374] LustreError: 254321:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6345.615526] Lustre: server umount lustre-OST0001 complete [ 6347.740833] LustreError: 253026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6347.759025] LustreError: 253026:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6361.059662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6361.063307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6361.068056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6361.080548] Lustre: Skipped 2 previous similar messages [ 6361.089074] Lustre: Skipped 1 previous similar message [ 6364.130946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6364.133546] LustreError: 252563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6364.146666] LustreError: 252563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6369.259288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6369.267823] Lustre: Skipped 2 previous similar messages [ 6375.391148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6375.464136] LustreError: 254545:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6375.473119] LustreError: 254545:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6375.594880] Lustre: server umount lustre-OST0000 complete [ 6379.490821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6379.494516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6379.498774] Lustre: Skipped 1 previous similar message [ 6379.506913] Lustre: Skipped 2 previous similar messages [ 6385.548095] Lustre: server umount lustre-MDT0000 complete [ 6389.129175] LustreError: 250821:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350502 with bad export cookie 900433572205048515 [ 6389.131308] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6389.158707] LustreError: 250821:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6395.571328] Lustre: server umount lustre-MDT0001 complete [ 6407.600538] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 6410.658965] Key type lgssc unregistered [ 6411.001582] LNet: 255483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6411.017374] LNetError: 255483:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6411.029044] LNet: Removed LNI 192.168.204.127@tcp [ 6411.930604] Key type .llcrypt unregistered [ 6411.932551] Key type ._llcrypt unregistered [ 6445.255625] Key type ._llcrypt registered [ 6445.257357] Key type .llcrypt registered [ 6445.366980] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 6458.788139] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 6459.958217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6459.995511] alg: No test for adler32 (adler32-zlib) [ 6461.088584] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 6461.390387] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 6463.095381] Key type lgssc registered [ 6464.137702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6472.690696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6479.347799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6485.976488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6491.921707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6498.394603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6498.476747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6498.504175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6499.857526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6499.886818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6499.945763] Lustre: lustre-MDT0000: new disk, initializing [ 6500.092179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6500.134754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6504.309391] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6514.334211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6514.380891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6514.443287] Lustre: 259447:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6514.489599] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6514.492106] Lustre: Skipped 1 previous similar message [ 6514.543630] Lustre: lustre-MDT0001: new disk, initializing [ 6514.618300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6514.668394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6514.684072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6518.634389] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6525.699025] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6529.359451] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6535.497887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6535.561684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6535.777887] Lustre: lustre-OST0000: new disk, initializing [ 6535.785151] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6535.887224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6541.667737] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6544.952115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6544.971313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6545.090522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6549.938269] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6553.923735] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6554.193475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6558.040263] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6558.274299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6560.739106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6560.740626] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6560.746372] LustreError: Skipped 1 previous similar message [ 6560.751976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6560.768376] Lustre: Skipped 1 previous similar message [ 6565.863083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6565.867166] Lustre: Skipped 1 previous similar message [ 6566.088199] LustreError: 262386:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6566.203577] Lustre: server umount lustre-OST0000 complete [ 6570.977432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6570.978367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6570.983214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6570.996451] Lustre: Skipped 2 previous similar messages [ 6575.541890] LustreError: 262588:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6575.549916] LustreError: 262588:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6575.652180] Lustre: server umount lustre-MDT0000 complete [ 6578.727100] LustreError: 261192:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350692 with bad export cookie 5023227112857095749 [ 6578.733191] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6578.747282] LustreError: 261192:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6578.907451] LustreError: 262789:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6578.922827] LustreError: 262789:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6579.132058] Lustre: server umount lustre-MDT0001 complete [ 6586.732530] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 22:38:19 (1763350699) [ 6593.677576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.074631] LustreError: 263461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6594.107884] LustreError: 263461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6594.165588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6597.662918] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6599.650143] LustreError: 263461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6603.757116] LustreError: 263461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6605.172636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6605.465875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6609.049509] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6613.618109] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6617.221613] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6623.344511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6623.698302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6629.411983] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6634.822102] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6690.272812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6690.276878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6690.278538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6690.282540] Lustre: Skipped 1 previous similar message [ 6690.287892] Lustre: Skipped 2 previous similar messages [ 6694.397868] LustreError: 265785:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6694.400632] LustreError: 265785:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6694.512528] Lustre: server umount lustre-OST0000 complete [ 6699.547678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6712.109345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6712.491752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6719.068840] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6725.248860] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6729.898151] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6729.903422] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6729.925121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6766.051298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6766.078822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6766.082539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6771.080151] LustreError: 267449:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6771.082834] LustreError: 267449:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6771.173474] LustreError: 267289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6771.275710] Lustre: server umount lustre-OST0000 complete [ 6776.296458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6776.298405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6776.305112] Lustre: Skipped 2 previous similar messages [ 6776.329356] Lustre: Skipped 1 previous similar message [ 6780.806702] LustreError: 267653:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6780.825638] LustreError: 267653:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6780.952907] Lustre: server umount lustre-MDT0000 complete [ 6781.409212] LustreError: 263461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6781.431272] LustreError: 263461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6785.084552] LustreError: 263441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350898 with bad export cookie 5023227112857096624 [ 6785.089063] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6785.110221] LustreError: 263441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6785.444177] Lustre: server umount lustre-MDT0001 complete [ 6796.757431] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 6799.791547] Key type lgssc unregistered [ 6800.104298] LNet: 268388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6800.112168] LNetError: 268388:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6800.128651] LNet: Removed LNI 192.168.204.127@tcp [ 6800.876494] Key type .llcrypt unregistered [ 6800.878816] Key type ._llcrypt unregistered [ 6821.927119] Key type ._llcrypt registered [ 6821.928399] Key type .llcrypt registered [ 6822.039705] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 6823.183888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6823.326476] alg: No test for adler32 (adler32-zlib) [ 6824.407085] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 6824.700717] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 6826.384313] Key type lgssc registered [ 6827.481711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6834.759783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6834.780132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6836.271829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6839.822386] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6846.438799] LustreError: 269777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6846.455866] LustreError: 269777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6848.067651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6848.542755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6851.992751] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6857.144417] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6860.884827] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6868.145755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6868.621796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6875.233690] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6880.761088] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6880.960921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6904.802052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6904.816547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6904.838320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6906.856198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6906.859323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6906.873113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6906.899775] Lustre: Skipped 1 previous similar message [ 6908.807597] LustreError: 272798:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6909.010649] Lustre: server umount lustre-OST0000 complete [ 6915.040370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6915.044782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6915.052308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6917.095822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6917.098303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6917.109021] Lustre: Skipped 1 previous similar message [ 6918.271497] LustreError: 272999:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6918.283380] LustreError: 272999:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6918.449985] Lustre: server umount lustre-MDT0000 complete [ 6921.343991] LustreError: 269759:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351034 with bad export cookie 14256942702930632931 [ 6921.351814] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6921.358564] LustreError: 269759:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6921.489814] LustreError: 273200:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6921.494260] LustreError: 273200:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6921.589804] Lustre: server umount lustre-MDT0001 complete [ 6931.042519] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 6933.253248] Key type lgssc unregistered [ 6933.516485] LNet: 273734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6933.524914] LNetError: 273734:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6933.547639] LNet: Removed LNI 192.168.204.127@tcp [ 6934.371233] Key type .llcrypt unregistered [ 6934.373208] Key type ._llcrypt unregistered [ 6949.879490] Key type ._llcrypt registered [ 6949.881783] Key type .llcrypt registered [ 6950.024960] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 6951.298659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6951.307861] alg: No test for adler32 (adler32-zlib) [ 6952.375684] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 6952.632406] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 6954.287331] Key type lgssc registered [ 6955.445879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6962.492094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6962.504783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6963.934312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6967.817394] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6973.486116] LustreError: 274940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6973.511361] LustreError: 274940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6976.184231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6976.576168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6979.736965] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 6984.602787] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6988.366399] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6995.557964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6995.879474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7000.483697] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7005.348733] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7009.210299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7022.560055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7022.571231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7022.593205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7024.613993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7024.616202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7024.626393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7025.597276] LustreError: 277331:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7025.746487] Lustre: server umount lustre-OST0000 complete [ 7029.729291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7029.730624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7029.740420] Lustre: Skipped 1 previous similar message [ 7029.749469] Lustre: Skipped 2 previous similar messages [ 7034.860712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7035.251749] LustreError: 277532:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7035.258017] LustreError: 277532:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7035.380043] Lustre: server umount lustre-MDT0000 complete [ 7038.752856] LustreError: 275659:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351152 with bad export cookie 6807196522585209964 [ 7038.757847] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7038.768680] LustreError: 275659:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7038.920716] LustreError: 277734:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7038.927284] LustreError: 277734:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7039.143951] Lustre: server umount lustre-MDT0001 complete [ 7048.766498] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 7051.479825] Key type lgssc unregistered [ 7051.726592] LNet: 278264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7051.735744] LNetError: 278264:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7051.751524] LNet: Removed LNI 192.168.204.127@tcp [ 7052.507730] Key type .llcrypt unregistered [ 7052.509479] Key type ._llcrypt unregistered [ 7072.718610] Key type ._llcrypt registered [ 7072.723062] Key type .llcrypt registered [ 7072.821881] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7073.533600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7073.619097] alg: No test for adler32 (adler32-zlib) [ 7074.656544] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 7074.837499] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 7076.519200] Key type lgssc registered [ 7077.772873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7085.180087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7085.216684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7086.723494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7090.200450] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7097.313516] LustreError: 279662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7097.339073] LustreError: 279662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7098.217445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7098.588393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7101.871885] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7106.281397] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7109.540231] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7114.832282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7115.149871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7120.217786] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7125.476401] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7131.561338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7150.049859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7150.054540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7150.067784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7152.099535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7152.102208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7155.963552] LustreError: 282679:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7156.112693] Lustre: server umount lustre-OST0000 complete [ 7160.289585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7160.293208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7160.301906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7160.313473] Lustre: Skipped 1 previous similar message [ 7162.341407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7162.348961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7162.374032] Lustre: Skipped 1 previous similar message [ 7165.834857] LustreError: 282881:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7165.850554] LustreError: 282881:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7165.943582] Lustre: server umount lustre-MDT0000 complete [ 7167.458153] LustreError: 279662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7167.469434] LustreError: 279662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7169.734827] LustreError: 279642:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351283 with bad export cookie 16731853831120310157 [ 7169.738837] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7169.740693] LustreError: 279642:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7169.862403] LustreError: 283084:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7169.878725] LustreError: 283084:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7170.127424] Lustre: server umount lustre-MDT0001 complete [ 7179.471488] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 7182.397234] Key type lgssc unregistered [ 7182.808587] LNet: 283616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7182.826517] LNetError: 283616:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7183.845784] LNet: Removed LNI 192.168.204.127@tcp [ 7185.006995] Key type .llcrypt unregistered [ 7185.008862] Key type ._llcrypt unregistered [ 7202.432647] Key type ._llcrypt registered [ 7202.435200] Key type .llcrypt registered [ 7202.584433] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7203.849524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7203.877146] alg: No test for adler32 (adler32-zlib) [ 7205.179220] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 7205.523212] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 7207.231216] Key type lgssc registered [ 7208.485853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7216.395664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7216.422756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7217.965936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7222.050775] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7228.387109] LustreError: 284844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7228.404382] LustreError: 284844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7230.653994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7231.122796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7235.059301] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7239.855827] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7242.795215] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7248.467697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7248.780138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7252.963762] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7257.646350] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7263.082219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7271.394440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7271.397061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7271.413239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7273.443611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7273.449727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7273.456847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7273.465144] Lustre: Skipped 1 previous similar message [ 7276.997534] LustreError: 287249:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7277.168587] Lustre: server umount lustre-OST0000 complete [ 7282.146382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7282.150062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7282.169210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7286.227056] LustreError: 287450:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7286.232370] LustreError: 287450:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7286.320178] Lustre: server umount lustre-MDT0000 complete [ 7288.803170] LustreError: 284826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7288.812504] LustreError: 284826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7289.268852] LustreError: 284799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351402 with bad export cookie 3736781463400046845 [ 7289.274603] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7289.280134] LustreError: 284799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7289.394875] LustreError: 287653:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7289.397090] LustreError: 287653:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7289.570443] Lustre: server umount lustre-MDT0001 complete [ 7298.263168] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 7300.591813] Key type lgssc unregistered [ 7300.859043] LNet: 288184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7300.868438] LNetError: 288184:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7300.881830] LNet: Removed LNI 192.168.204.127@tcp [ 7301.535605] Key type .llcrypt unregistered [ 7301.537114] Key type ._llcrypt unregistered [ 7310.841835] Key type ._llcrypt registered [ 7310.843576] Key type .llcrypt registered [ 7310.931380] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7322.817789] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 7333.534029] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7334.013152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7334.026535] alg: No test for adler32 (adler32-zlib) [ 7334.996136] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 7335.154656] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 7336.791198] Key type lgssc registered [ 7337.610752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7343.406753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7347.832211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7353.334162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7358.272898] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7364.409328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7364.468438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7364.480707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7365.722641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7365.777367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7365.836921] Lustre: lustre-MDT0000: new disk, initializing [ 7365.989393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7366.019552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7370.032892] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7378.876782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7378.928209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7379.009107] Lustre: 291670:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7379.077099] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7379.081457] Lustre: Skipped 1 previous similar message [ 7379.125483] Lustre: lustre-MDT0001: new disk, initializing [ 7379.205509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7379.248109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7379.266477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7382.217924] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7387.751602] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7390.646895] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7395.611903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7395.664214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7395.845691] Lustre: lustre-OST0000: new disk, initializing [ 7395.848650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7395.941932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7400.507764] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7403.082805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7403.103162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7403.198199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7406.493583] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7410.545648] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7410.896890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7414.280528] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7414.492449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7418.850497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7418.850944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7418.853711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7418.859762] LustreError: Skipped 1 previous similar message [ 7418.875184] Lustre: Skipped 1 previous similar message [ 7423.981445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7423.988017] Lustre: Skipped 2 previous similar messages [ 7429.095666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7430.111439] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7430.146519] LustreError: 294611:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7430.238829] Lustre: server umount lustre-OST0000 complete [ 7434.209310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7434.210163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7434.221902] Lustre: Skipped 1 previous similar message [ 7434.225121] Lustre: Skipped 1 previous similar message [ 7439.330906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7439.335457] Lustre: Skipped 2 previous similar messages [ 7439.344103] LustreError: 294815:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7439.346373] LustreError: 294815:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7439.421188] Lustre: server umount lustre-MDT0000 complete [ 7442.299498] LustreError: 291662:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351555 with bad export cookie 14690736599066868976 [ 7442.305097] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7442.309207] LustreError: 291662:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7442.439386] LustreError: 295016:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7442.441968] LustreError: 295016:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7442.566248] Lustre: server umount lustre-MDT0001 complete [ 7449.901286] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 22:52:42 (1763351562) [ 7455.289281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7455.742180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7458.810728] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7460.833808] LustreError: 295686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7460.849500] LustreError: 295686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7465.460163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7465.715364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7468.362365] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7473.176389] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7476.449231] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7482.387458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7482.636373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7487.610409] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7492.386938] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7511.523173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7511.531202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7511.540072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7512.551826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7514.688115] LustreError: 297955:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7514.695188] LustreError: 297955:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7514.834836] Lustre: server umount lustre-OST0000 complete [ 7522.272769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7522.293195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7522.323943] Lustre: Skipped 1 previous similar message [ 7524.281761] LustreError: 298156:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7524.300809] LustreError: 298156:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7524.456466] Lustre: server umount lustre-MDT0000 complete [ 7527.906883] LustreError: 295687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7527.923292] LustreError: 295687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7528.682458] LustreError: 295667:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351642 with bad export cookie 14690736599066869851 [ 7528.686832] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7528.692428] LustreError: 295667:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7529.291682] Lustre: server umount lustre-MDT0001 complete [ 7539.985494] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 7542.263830] Key type lgssc unregistered [ 7542.520611] LNet: 298892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7542.524597] LNetError: 298892:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7542.540859] LNet: Removed LNI 192.168.204.127@tcp [ 7543.233379] Key type .llcrypt unregistered [ 7543.235815] Key type ._llcrypt unregistered [ 7567.817800] Key type ._llcrypt registered [ 7567.819641] Key type .llcrypt registered [ 7567.906906] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 7579.647698] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7580.369677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7580.396779] alg: No test for adler32 (adler32-zlib) [ 7581.427949] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 7581.596199] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 7583.248428] Key type lgssc registered [ 7584.258757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7591.735848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7597.000605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7602.864980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7609.153384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7614.046461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7619.451647] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7625.886291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7625.936712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7625.952448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7627.189648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7627.221099] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7627.271139] Lustre: lustre-MDT0000: new disk, initializing [ 7627.387886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7627.412710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7630.852948] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7639.392685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7639.449542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7639.515907] Lustre: 302680:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7639.585955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7639.592963] Lustre: Skipped 1 previous similar message [ 7639.696725] Lustre: lustre-MDT0001: new disk, initializing [ 7639.812705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7639.847335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7639.854911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7643.299959] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7650.064889] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7653.361405] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7659.256815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7659.314612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7659.496641] Lustre: lustre-OST2710: new disk, initializing [ 7659.503578] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7659.587090] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7664.074392] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7666.226790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7666.238027] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7666.331110] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7670.763699] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7676.895281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7676.963969] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7677.088236] Lustre: lustre-OST03e8: new disk, initializing [ 7677.095291] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7677.183600] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7682.815482] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7687.264753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7687.279569] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7687.401816] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7687.809795] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7694.393416] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7694.563331] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7698.407307] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7698.656774] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7701.888265] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7702.112475] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7705.627910] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7705.808626] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7711.714588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7711.720654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7711.732773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7713.253577] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7713.254736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7713.262367] Lustre: Skipped 2 previous similar messages [ 7713.265669] Lustre: Skipped 2 previous similar messages [ 7715.027982] LustreError: 307023:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7715.118647] Lustre: server umount lustre-MDT0000 complete [ 7717.969258] LustreError: 302671:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351831 with bad export cookie 11021854533063528612 [ 7717.970652] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7717.973335] LustreError: 302671:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7718.111175] LustreError: 307225:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7718.114774] LustreError: 307225:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7718.257126] Lustre: server umount lustre-MDT0001 complete [ 7730.629157] LustreError: 307428:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7730.631810] LustreError: 307428:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7730.706071] Lustre: server umount lustre-OST2710 complete [ 7743.812125] LustreError: 307630:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7743.815342] LustreError: 307630:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7743.971656] Lustre: server umount lustre-OST03e8 complete [ 7757.067718] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 7764.685729] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7772.708545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7778.018793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7782.500367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7786.802928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7791.667041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7791.743990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7791.918918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7791.948332] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7792.013931] Lustre: lustre-MDT0000: new disk, initializing [ 7792.146529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7792.177067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7795.335858] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7802.402920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7802.442727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7802.501621] Lustre: 310426:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7802.540233] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7802.543251] Lustre: Skipped 1 previous similar message [ 7802.584646] Lustre: lustre-MDT0001: new disk, initializing [ 7802.644377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7802.677043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7802.681120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7804.968617] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7811.220963] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7814.069980] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7819.468585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7819.538652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7819.736114] Lustre: lustre-OST0000: new disk, initializing [ 7819.740352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7819.825825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7824.075374] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7829.526193] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7829.555456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7829.564819] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7829.645429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7832.266274] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7832.434822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7835.589166] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7835.796505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7840.224854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7840.227464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7840.230089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7840.239298] Lustre: Skipped 1 previous similar message [ 7843.457640] LustreError: 313361:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7843.464547] LustreError: 313361:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7843.563388] Lustre: server umount lustre-OST0000 complete [ 7848.931892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7848.938742] LustreError: Skipped 1 previous similar message [ 7848.948734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7848.961178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7848.964059] Lustre: Skipped 1 previous similar message [ 7853.176613] LustreError: 313563:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7853.184266] LustreError: 313563:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7853.265910] Lustre: server umount lustre-MDT0000 complete [ 7856.230588] LustreError: 310417:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351969 with bad export cookie 11021854533063530292 [ 7856.236877] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7856.245151] LustreError: 310417:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7856.501905] Lustre: server umount lustre-MDT0001 complete [ 7862.333798] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 22:59:35 (1763351975) [ 7863.573865] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7864.850741] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 22:59:37 (1763351977) [ 7867.123380] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7872.829810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7875.962597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7879.010369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7882.198035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7883.116601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7888.580594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7888.749358] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7888.781343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7888.920709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7891.525366] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7897.882678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7897.939835] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7897.972756] Lustre: 315893:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7900.899856] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7904.893666] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7907.287616] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7912.056030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7912.100779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7912.201648] Lustre: Denying initial registration attempt from nid 192.168.204.127@tcp, specified as failover [ 7912.210518] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7912.215967] LustreError: 317507:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 7912.219986] LustreError: 317507:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 7912.224269] LustreError: 317507:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 7912.274704] Lustre: server umount lustre-OST0000 complete [ 7912.277711] LustreError: 317507:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7918.561936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7918.566088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7918.573959] Lustre: Skipped 1 previous similar message [ 7918.577836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7918.581859] Lustre: Skipped 2 previous similar messages [ 7929.314518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7929.316346] Lustre: Skipped 5 previous similar messages [ 7931.359136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7931.428566] LustreError: 317876:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 7931.441925] LustreError: 317876:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7931.540055] Lustre: server umount lustre-MDT0000 complete [ 7933.853931] LustreError: 315884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352047 with bad export cookie 11021854533063531174 [ 7933.856412] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7933.865576] LustreError: 315884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7934.126811] Lustre: server umount lustre-MDT0001 complete [ 7940.245412] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 7941.881359] Key type lgssc unregistered [ 7942.064671] LNet: 318612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7942.068327] LNetError: 318612:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7942.083058] LNet: Removed LNI 192.168.204.127@tcp [ 7942.643659] Key type .llcrypt unregistered [ 7942.649371] Key type ._llcrypt unregistered [ 7950.298735] Key type ._llcrypt registered [ 7950.300113] Key type .llcrypt registered [ 7950.399355] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7951.140153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7951.147865] alg: No test for adler32 (adler32-zlib) [ 7952.158051] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 7952.333571] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 7954.010535] Key type lgssc registered [ 7954.922646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7959.727563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7961.990089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7964.609695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7967.739157] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7968.594021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7978.817601] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7984.791229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7984.805296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7985.985331] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7986.006426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7986.125339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7988.955196] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7995.111932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7995.183568] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7995.217776] Lustre: 321356:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7995.413948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7998.209140] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8002.049321] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8004.620152] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8009.047189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8009.105582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8009.252559] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8009.392392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8012.894754] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8016.329317] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8021.473658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8021.475125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8021.476955] LustreError: Skipped 1 previous similar message [ 8021.484974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8021.490762] Lustre: Skipped 1 previous similar message [ 8023.738318] LustreError: 323640:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8023.825546] Lustre: server umount lustre-OST0000 complete [ 8026.593226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8026.599995] Lustre: Skipped 1 previous similar message [ 8026.607078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8031.201143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8031.213615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8031.222228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8032.609520] LustreError: 323841:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8032.616018] LustreError: 323841:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8032.705659] Lustre: server umount lustre-MDT0000 complete [ 8035.952789] LustreError: 322083:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352149 with bad export cookie 16454527535262406737 [ 8035.954259] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8035.959295] LustreError: 322083:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8036.064191] LustreError: 324044:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8036.091784] LustreError: 324044:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8036.311486] Lustre: server umount lustre-MDT0001 complete [ 8042.800677] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8044.510157] Key type lgssc unregistered [ 8044.729619] LNet: 324575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8044.735481] LNetError: 324575:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8044.752909] LNet: Removed LNI 192.168.204.127@tcp [ 8045.329105] Key type .llcrypt unregistered [ 8045.331066] Key type ._llcrypt unregistered [ 8061.689617] Key type ._llcrypt registered [ 8061.691725] Key type .llcrypt registered [ 8061.765824] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8062.373302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8062.402342] alg: No test for adler32 (adler32-zlib) [ 8063.361120] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 8063.576743] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 8065.208547] Key type lgssc registered [ 8066.118302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8071.392956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8071.402935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8072.713981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8075.259899] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8081.329097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8081.599354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8084.442822] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8088.082329] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8090.648684] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8095.135072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8095.403297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8098.981813] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8102.083036] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8114.648621] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.27@tcp (stopping) [ 8116.422174] LustreError: 328223:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8116.485856] Lustre: server umount lustre-MDT0000 complete [ 8117.216368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8117.223420] LustreError: 326663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8117.236021] LustreError: 326663:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8119.107713] LustreError: 325930:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352232 with bad export cookie 13995338710744371693 [ 8119.108655] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8119.119618] LustreError: 325930:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8119.762598] LustreError: 327734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8119.789666] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.27@tcp (stopping) [ 8122.341579] LustreError: 325945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8122.342647] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8122.381217] Lustre: Skipped 2 previous similar messages [ 8122.393046] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8124.387312] LustreError: 325945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8124.874986] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.27@tcp (stopping) [ 8124.878381] Lustre: Skipped 1 previous similar message [ 8125.304421] LustreError: 328425:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8125.308780] LustreError: 328425:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8125.468213] Lustre: server umount lustre-MDT0001 complete [ 8130.385583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8136.378863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8143.840125] LustreError: 325259:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000047e9c456 x1849008772376832/t0(0) o250->MGC192.168.204.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8144.047897] LustreError: 327559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8144.063854] LustreError: 327559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8144.110348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8144.155939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8146.524153] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8151.760683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8152.101333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8154.155885] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8154.785550] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8158.241960] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8159.245549] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8161.471689] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8162.562370] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8167.392978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8167.406097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8167.422803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8169.447424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8171.385196] LustreError: 330729:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8171.388407] LustreError: 330729:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8171.441395] Lustre: server umount lustre-OST0000 complete [ 8174.561238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8174.569111] Lustre: Skipped 1 previous similar message [ 8179.590246] LustreError: 330930:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8179.592654] LustreError: 330930:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8179.651668] Lustre: server umount lustre-MDT0000 complete [ 8179.680494] LustreError: 329236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8179.689059] LustreError: 329236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8181.609823] LustreError: 325932:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352295 with bad export cookie 13995338710744378826 [ 8181.612487] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8181.616754] LustreError: 325932:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8181.769811] Lustre: server umount lustre-MDT0001 complete [ 8186.985360] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8188.339908] Key type lgssc unregistered [ 8188.498761] LNet: 331662:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8188.503411] LNetError: 331662:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8188.513370] LNet: Removed LNI 192.168.204.127@tcp [ 8188.886813] Key type .llcrypt unregistered [ 8188.888307] Key type ._llcrypt unregistered [ 8203.257758] Key type ._llcrypt registered [ 8203.259422] Key type .llcrypt registered [ 8203.316752] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8203.754413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8203.779479] alg: No test for adler32 (adler32-zlib) [ 8204.724833] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 8204.863526] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 8206.479162] Key type lgssc registered [ 8207.290735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8211.561860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8211.572956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8212.832396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8214.651751] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8218.579307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8218.806448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8220.747269] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8223.370593] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8225.039808] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8228.324460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8228.326574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8228.329936] Lustre: Skipped 1 previous similar message [ 8228.334810] Lustre: Skipped 1 previous similar message [ 8229.345056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8232.778120] LustreError: 334427:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8232.820849] Lustre: server umount lustre-MDT0000 complete [ 8234.464992] LustreError: 333036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8234.471945] LustreError: 333036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8234.501319] LustreError: 334428:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352348 with bad export cookie 15802167822961392376 [ 8234.504350] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8234.506025] LustreError: 334428:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8234.575637] LustreError: 334629:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8234.578423] LustreError: 334629:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8234.652528] Lustre: server umount lustre-MDT0001 complete [ 8241.352117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8241.506475] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8241.527726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8241.637371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8243.450108] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8247.571367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8247.635935] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8247.650391] Lustre: 335399:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8247.790829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8249.574125] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8252.234523] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8254.121104] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8258.209681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8258.323501] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8258.331577] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8258.336755] LustreError: 337057:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 8258.341518] LustreError: 337057:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 8258.344554] LustreError: 337057:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 8258.383128] Lustre: server umount lustre-OST0000 complete [ 8258.384849] LustreError: 337057:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8263.228390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8263.376458] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8263.503573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8266.660428] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8269.706201] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8270.929563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8274.630199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8274.681716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8274.715102] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8274.766740] Lustre: lustre-OST0001: new disk, initializing [ 8274.769856] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8274.830092] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8276.062621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8276.067666] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8276.108123] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8277.874894] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8280.948222] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8286.689562] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8286.691899] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8286.696485] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8286.702367] LustreError: Skipped 1 previous similar message [ 8286.705347] Lustre: Skipped 2 previous similar messages [ 8288.567144] LustreError: 339179:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 8288.571751] LustreError: 339179:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8288.599704] Lustre: server umount lustre-OST0001 complete [ 8291.296254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8291.300111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8291.309330] Lustre: Skipped 1 previous similar message [ 8291.312955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8291.316933] Lustre: Skipped 1 previous similar message [ 8291.811821] LustreError: 337526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8291.820066] LustreError: 337526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8296.698471] LustreError: 339380:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8296.701100] LustreError: 339380:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8296.796791] Lustre: server umount lustre-OST0000 complete [ 8302.051673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8302.054178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8302.062055] Lustre: Skipped 2 previous similar messages [ 8305.369233] LustreError: 339581:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8305.372626] LustreError: 339581:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8305.437605] Lustre: server umount lustre-MDT0000 complete [ 8307.172578] LustreError: 335410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8307.183264] LustreError: 335410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8307.605589] LustreError: 336121:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352421 with bad export cookie 15802167822961392915 [ 8307.606116] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8307.611581] LustreError: 336121:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8307.858920] Lustre: server umount lustre-MDT0001 complete [ 8313.894608] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8315.444087] Key type lgssc unregistered [ 8315.611743] LNet: 340313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8315.618091] LNetError: 340313:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8315.631249] LNet: Removed LNI 192.168.204.127@tcp [ 8316.104362] Key type .llcrypt unregistered [ 8316.106971] Key type ._llcrypt unregistered [ 8334.751144] Key type ._llcrypt registered [ 8334.753264] Key type .llcrypt registered [ 8334.824709] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 8343.701475] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8344.241407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8344.256305] alg: No test for adler32 (adler32-zlib) [ 8345.182689] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 8345.320968] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 8346.951201] Key type lgssc registered [ 8347.551644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8351.904441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8355.300947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8358.358932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8362.276579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8365.843581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8365.872681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8365.880596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8367.020995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8367.044727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8367.090119] Lustre: lustre-MDT0000: new disk, initializing [ 8367.128746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8367.139361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8369.168764] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8375.447178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8375.494968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8375.555876] Lustre: 344239:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8375.589575] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8375.591853] Lustre: Skipped 1 previous similar message [ 8375.647207] Lustre: lustre-MDT0001: new disk, initializing [ 8375.720276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8375.745028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8375.756368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8377.896788] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8382.417448] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8384.323253] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8387.823231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8387.856256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8387.997914] Lustre: lustre-OST0000: new disk, initializing [ 8388.002816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8388.049420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8390.712212] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8391.503313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8391.507461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8391.561314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8394.770653] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8396.596359] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8396.696947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8398.567788] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8398.661920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8401.891501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8401.892196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8401.894440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8401.897444] LustreError: Skipped 1 previous similar message [ 8401.902988] Lustre: Skipped 1 previous similar message [ 8405.717304] LustreError: 347177:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8405.799057] Lustre: server umount lustre-OST0000 complete [ 8411.616953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8411.620328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8411.629770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8411.632611] Lustre: Skipped 1 previous similar message [ 8414.735879] LustreError: 347379:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8414.740016] LustreError: 347379:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8414.791352] Lustre: server umount lustre-MDT0000 complete [ 8416.786156] LustreError: 345985:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352530 with bad export cookie 11757921867452505046 [ 8416.791363] LustreError: 345985:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8416.792602] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8416.896953] LustreError: 347582:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8416.900634] LustreError: 347582:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8416.982119] Lustre: server umount lustre-MDT0001 complete [ 8421.290676] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 23:08:54 (1763352534) [ 8424.147473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8427.513862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8439.341873] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 8443.886386] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8449.557228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8453.531984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8456.804631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8460.195797] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8463.433653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8463.462742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8463.571372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8463.588383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8463.631583] Lustre: lustre-MDT0000: new disk, initializing [ 8463.677213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8463.686514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8465.616450] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8471.745359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8471.780489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8471.835624] Lustre: 351351:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8471.872265] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8471.875461] Lustre: Skipped 1 previous similar message [ 8471.920846] Lustre: lustre-MDT0001: new disk, initializing [ 8471.970775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8471.987235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8471.991850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8473.999744] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8478.787757] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8480.805201] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8484.114792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8484.163409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8484.303855] Lustre: lustre-OST0000: new disk, initializing [ 8484.307357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8484.353532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8487.397772] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8490.537507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8490.542054] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8490.605106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8491.753334] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8493.998708] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8494.107701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8496.202625] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8496.310416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8500.705287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8500.706777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8500.710503] LustreError: Skipped 1 previous similar message [ 8500.720853] Lustre: Skipped 1 previous similar message [ 8500.723273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8500.729196] Lustre: Skipped 3 previous similar messages [ 8503.476070] LustreError: 354288:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8503.478725] LustreError: 354288:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8503.541952] Lustre: server umount lustre-OST0000 complete [ 8505.825549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8505.828064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8505.832918] Lustre: Skipped 2 previous similar messages [ 8510.945805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8510.953610] Lustre: Skipped 2 previous similar messages [ 8511.631713] LustreError: 354489:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8511.633948] LustreError: 354489:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8511.677201] Lustre: server umount lustre-MDT0000 complete [ 8513.483959] LustreError: 351344:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352627 with bad export cookie 11757921867452505935 [ 8513.489673] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8513.491505] LustreError: 351344:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8513.687510] Lustre: server umount lustre-MDT0001 complete [ 8517.694811] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 23:10:31 (1763352631) [ 8521.687708] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 23:10:35 (1763352635) [ 8526.279816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8526.603663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8528.771382] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8531.939063] LustreError: 355703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8531.947992] LustreError: 355703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8533.475073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8535.811341] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8538.936394] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8541.124159] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8544.871403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8548.079134] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8550.910730] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8565.507122] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8566.538455] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8567.450591] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8568.437881] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 8569.331228] Lustre: Failing over lustre-MDT0000 [ 8569.414447] LustreError: 358160:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8569.416625] LustreError: 358160:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8569.457133] Lustre: server umount lustre-MDT0000 complete [ 8569.825751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8569.826202] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8569.828685] LustreError: 355699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8569.841709] Lustre: Skipped 2 previous similar messages [ 8572.889295] LustreError: 355697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8572.897140] LustreError: 355697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8574.950387] LustreError: 355698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8574.964690] LustreError: 355698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8580.065326] LustreError: 355702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8580.072425] LustreError: 355702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8583.857053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8583.930686] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8584.061271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8584.064522] Lustre: Skipped 2 previous similar messages [ 8584.097064] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8585.943794] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8588.200879] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8589.290574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8589.300848] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 8589.322835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8591.564339] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8592.710574] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8595.472694] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 8596.575658] Lustre: Failing over lustre-MDT0000 [ 8596.686582] LustreError: 359239:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8596.688613] LustreError: 359239:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8596.731379] Lustre: server umount lustre-MDT0000 complete [ 8598.489507] LustreError: 355697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8598.499160] LustreError: 355697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8599.522534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8599.523173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8599.527801] Lustre: Skipped 2 previous similar messages [ 8610.985079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8611.067762] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8611.243806] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8613.151966] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8613.834206] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8616.422266] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8616.426028] Lustre: Skipped 2 previous similar messages [ 8616.437640] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 8616.466065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8618.257596] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8619.038378] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8626.658095] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8626.659098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8626.664834] Lustre: Skipped 2 previous similar messages [ 8626.668135] Lustre: Skipped 1 previous similar message [ 8626.670535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8630.178313] LustreError: 360267:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8630.180446] LustreError: 360267:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 8630.202119] Lustre: server umount lustre-MDT0000 complete [ 8631.777587] LustreError: 355703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8631.790037] LustreError: 355703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 8632.060859] LustreError: 355683:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352745 with bad export cookie 11757921867452508602 [ 8632.064921] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8632.067361] LustreError: 355683:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8632.245962] Lustre: server umount lustre-MDT0001 complete [ 8644.499550] Lustre: server umount lustre-OST0000 complete [ 8652.594815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8652.916544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8652.919476] Lustre: Skipped 1 previous similar message [ 8655.076818] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8660.104982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8662.771423] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8666.428139] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8668.967372] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8673.286369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8677.078913] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8678.644898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8680.537444] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8689.121573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8689.124991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8689.134698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8694.077338] Lustre: server umount lustre-OST0000 complete [ 8702.105984] LustreError: 363945:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8702.109412] LustreError: 363945:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8702.161806] Lustre: server umount lustre-MDT0000 complete [ 8704.217127] LustreError: 361480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352817 with bad export cookie 11757921867452509337 [ 8704.220729] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8704.224941] LustreError: 361480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8704.426779] Lustre: server umount lustre-MDT0001 complete [ 8709.964858] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8711.307319] Key type lgssc unregistered [ 8711.466180] LNet: 364680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8711.470914] LNetError: 364680:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8711.485579] LNet: Removed LNI 192.168.204.127@tcp [ 8711.865464] Key type .llcrypt unregistered [ 8711.867664] Key type ._llcrypt unregistered [ 8724.667824] Key type ._llcrypt registered [ 8724.671378] Key type .llcrypt registered [ 8724.729668] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8725.257664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8725.274249] alg: No test for adler32 (adler32-zlib) [ 8726.173337] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 8726.311466] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 8727.911661] Key type lgssc registered [ 8728.534376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8732.366411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8732.376796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8733.609184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8735.342605] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8739.554573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8739.772195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8741.707588] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8744.660366] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8746.548156] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8749.621631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8749.787840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8752.189982] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8754.705595] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8759.019966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8769.505791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8769.506667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8769.514335] Lustre: Skipped 2 previous similar messages [ 8769.520946] Lustre: Skipped 2 previous similar messages [ 8772.567940] LustreError: 368423:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8772.615155] Lustre: server umount lustre-MDT0000 complete [ 8774.540401] LustreError: 366138:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352888 with bad export cookie 7048650721639491801 [ 8774.544168] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8774.545611] LustreError: 366138:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8774.624257] LustreError: 367904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8774.624337] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8774.632466] LustreError: 367904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8774.639492] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8774.654140] LustreError: 368625:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8774.656315] LustreError: 368625:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8774.750364] Lustre: server umount lustre-MDT0001 complete [ 8782.840410] LustreError: 368825:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8782.844962] LustreError: 368825:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8782.906325] Lustre: server umount lustre-OST0000 complete [ 8789.854987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8790.071970] LustreError: 369554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8790.111671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8791.901890] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8795.616689] LustreError: 369554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8795.728229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8795.919951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8797.815480] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8800.421143] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8802.117312] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8805.613309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8805.784446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8808.466410] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8810.953476] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8815.020078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 8825.312698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8825.316176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8825.318185] Lustre: Skipped 2 previous similar messages [ 8825.326639] Lustre: Skipped 1 previous similar message [ 8830.154986] LustreError: 371898:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8830.156928] LustreError: 371898:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8830.200631] Lustre: server umount lustre-MDT0000 complete [ 8830.433245] LustreError: 369554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8830.440737] LustreError: 369554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8831.789826] LustreError: 369535:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352945 with bad export cookie 7048650721639492914 [ 8831.793466] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8831.796462] LustreError: 369535:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8831.932801] Lustre: server umount lustre-MDT0001 complete [ 8843.770626] LustreError: 372302:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8843.773836] LustreError: 372302:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 8843.855619] Lustre: server umount lustre-OST0000 complete [ 8852.004868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8852.227898] LustreError: 373130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8852.269789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8854.361386] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8858.992550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8861.029874] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8863.430680] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8865.234872] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8868.806885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8872.086895] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8874.725370] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8877.225582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 8882.658543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8882.659793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8882.662232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8882.668446] Lustre: Skipped 1 previous similar message [ 8882.676041] Lustre: Skipped 1 previous similar message [ 8883.893334] LustreError: 375375:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8883.895484] LustreError: 375375:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8883.950709] Lustre: server umount lustre-OST0000 complete [ 8887.777924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8887.783653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8887.788869] Lustre: Skipped 1 previous similar message [ 8887.882374] Lustre: server umount lustre-MDT0000 complete [ 8889.862736] LustreError: 375576:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353003 with bad export cookie 7048650721639493922 [ 8889.865993] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8889.867439] LustreError: 375576:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8890.021489] Lustre: server umount lustre-MDT0001 complete [ 8894.816502] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8896.157327] Key type lgssc unregistered [ 8896.303863] LNet: 376307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8896.308165] LNetError: 376307:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8896.318823] LNet: Removed LNI 192.168.204.127@tcp [ 8896.679839] Key type .llcrypt unregistered [ 8896.681668] Key type ._llcrypt unregistered [ 8908.330705] Key type ._llcrypt registered [ 8908.332367] Key type .llcrypt registered [ 8908.383229] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8908.865483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8908.877763] alg: No test for adler32 (adler32-zlib) [ 8909.768317] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 8909.889749] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 8911.495273] Key type lgssc registered [ 8912.090138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8915.768234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8915.777203] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8915.781783] LustreError: 377704:0:(osd_handler.c:8577:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8915.794295] LustreError: 377704:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8915.798125] LustreError: 377704:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8915.802087] LustreError: 377704:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8915.805944] LustreError: 377704:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8919.168027] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8919.172428] LustreError: 378102:0:(osd_handler.c:8577:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8919.179628] LustreError: 378102:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8919.183565] LustreError: 378102:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8919.186908] LustreError: 378102:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8919.189932] LustreError: 378102:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8927.720986] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8928.924364] Key type lgssc unregistered [ 8929.066847] LNet: 379086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8929.070681] LNetError: 379086:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8929.078689] LNet: Removed LNI 192.168.204.127@tcp [ 8929.424706] Key type .llcrypt unregistered [ 8929.426372] Key type ._llcrypt unregistered [ 8937.522059] Key type ._llcrypt registered [ 8937.523515] Key type .llcrypt registered [ 8937.572084] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 8944.875556] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8945.400909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8945.438566] alg: No test for adler32 (adler32-zlib) [ 8946.320661] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 8946.438132] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 8948.039332] Key type lgssc registered [ 8948.538537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8952.380107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8955.036928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8957.620840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8960.344682] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8963.268708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8963.293995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8963.302045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8964.409398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8964.424221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8964.465739] Lustre: lustre-MDT0000: new disk, initializing [ 8964.503893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8964.513312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8966.120972] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8971.117133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8971.146939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8971.192253] Lustre: 382192:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8971.227567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8971.230176] Lustre: Skipped 1 previous similar message [ 8971.277888] Lustre: lustre-MDT0001: new disk, initializing [ 8971.334086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8971.358511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8971.364893] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8973.046544] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8977.171589] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8978.891019] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8982.367782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8982.400830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8982.518543] Lustre: lustre-OST0000: new disk, initializing [ 8982.521497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8982.566729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8985.232180] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8989.259483] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8990.225436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8990.233961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8990.291721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8991.326553] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8991.437613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8993.593285] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8993.704851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8995.297336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8995.299464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8995.301703] LustreError: Skipped 1 previous similar message [ 8995.307542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8995.310880] Lustre: Skipped 1 previous similar message [ 8996.658517] LustreError: 385128:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8996.717238] Lustre: server umount lustre-OST0000 complete [ 9000.416967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9000.419735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9000.422833] Lustre: Skipped 2 previous similar messages [ 9004.494050] LustreError: 385329:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9004.497681] LustreError: 385329:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9004.551201] Lustre: server umount lustre-MDT0000 complete [ 9006.556406] LustreError: 382182:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353120 with bad export cookie 13675241712096920739 [ 9006.560572] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9006.562526] LustreError: 382182:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9006.657485] LustreError: 385531:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9006.660708] LustreError: 385531:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9006.745373] Lustre: server umount lustre-MDT0001 complete [ 9011.011496] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9011.843815] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 23:18:45 (1763353125) [ 9015.118983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9015.355698] LustreError: 386249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9015.365708] LustreError: 386249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9015.394583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9017.171579] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9020.897500] LustreError: 386249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9021.009229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9021.187182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9023.016332] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9025.369595] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9027.002769] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9030.143787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9030.319778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9032.862521] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9035.277566] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9038.527186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9038.564839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9038.634301] Lustre: lustre-OST0001: new disk, initializing [ 9038.637195] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9041.373241] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9042.382216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9042.387134] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9042.434252] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9045.375769] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9047.520780] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9047.524650] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9047.531179] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9047.533624] Lustre: Skipped 2 previous similar messages [ 9052.116350] Lustre: lustre-OST0001: Not available for connect from 192.168.204.27@tcp (stopping) [ 9053.619131] LustreError: 389447:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 9053.637716] LustreError: 389447:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9053.669557] Lustre: server umount lustre-OST0001 complete [ 9057.763548] LustreError: 387846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9057.770593] LustreError: 387846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9062.882318] LustreError: 387847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9062.888662] LustreError: 387847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9067.487869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9067.490976] LustreError: Skipped 1 previous similar message [ 9067.492722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9067.497603] Lustre: Skipped 1 previous similar message [ 9067.499885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9067.502813] Lustre: Skipped 2 previous similar messages [ 9068.001826] LustreError: 389567:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9068.007132] LustreError: 389567:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9072.365137] LustreError: 389674:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9072.367740] LustreError: 389674:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9072.415641] Lustre: server umount lustre-OST0000 complete [ 9077.728332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9077.731821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9077.737603] Lustre: Skipped 1 previous similar message [ 9077.740144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9077.742833] Lustre: Skipped 1 previous similar message [ 9080.180615] Lustre: server umount lustre-MDT0000 complete [ 9081.774343] LustreError: 386229:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353195 with bad export cookie 13675241712096921607 [ 9081.777463] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9081.779488] LustreError: 386229:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9096.160327] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9096.228665] LustreError: 390077:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 9096.232868] LustreError: 390077:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 9096.319264] Lustre: server umount lustre-MDT0001 complete [ 9100.857969] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 9102.084223] Key type lgssc unregistered [ 9102.227369] LNet: 390610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9102.230674] LNetError: 390610:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9102.237384] LNet: Removed LNI 192.168.204.127@tcp [ 9102.623838] Key type .llcrypt unregistered [ 9102.625724] Key type ._llcrypt unregistered [ 9116.841540] Key type ._llcrypt registered [ 9116.843113] Key type .llcrypt registered [ 9116.886780] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 9124.102439] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 9124.644329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9124.655174] alg: No test for adler32 (adler32-zlib) [ 9125.530607] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 9125.630655] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 9127.223256] Key type lgssc registered [ 9127.681737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9131.501609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9134.291540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9136.742272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9139.363717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9142.488747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9142.513340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9142.520427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9143.613187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9143.628216] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9143.670768] Lustre: lustre-MDT0000: new disk, initializing [ 9143.707435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9143.716434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9145.263174] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9150.574389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9150.610554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9150.660402] Lustre: 394535:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9150.691255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9150.694472] Lustre: Skipped 1 previous similar message [ 9150.737825] Lustre: lustre-MDT0001: new disk, initializing [ 9150.783037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9150.802245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9150.806100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9152.481283] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9156.444724] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9158.042966] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9161.244968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9161.280240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9161.410224] Lustre: lustre-OST0000: new disk, initializing [ 9161.414125] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9161.456042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9163.929458] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9167.638728] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9168.385557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9168.390144] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9168.436188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9169.407252] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9169.500689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9171.173771] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9171.263348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9173.474963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9173.475214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9173.479714] LustreError: Skipped 1 previous similar message [ 9173.488360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9173.493032] Lustre: Skipped 1 previous similar message [ 9174.131535] LustreError: 397472:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9174.187510] Lustre: server umount lustre-OST0000 complete [ 9176.544725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9176.548249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9176.554451] Lustre: Skipped 1 previous similar message [ 9176.558638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9178.592782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9178.595825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9183.712485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9183.715522] Lustre: Skipped 2 previous similar messages [ 9188.832146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9188.834811] Lustre: Skipped 1 previous similar message [ 9190.367534] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9190.406680] LustreError: 397673:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9190.408981] LustreError: 397673:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9190.442668] Lustre: server umount lustre-MDT0000 complete [ 9191.916478] LustreError: 394527:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353305 with bad export cookie 2308591593756112084 [ 9191.919235] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9191.921968] LustreError: 394527:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9191.990809] LustreError: 397876:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9191.993341] LustreError: 397876:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9192.058791] Lustre: server umount lustre-MDT0001 complete [ 9195.543996] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 23:21:49 (1763353309) [ 9199.498074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9202.759034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9202.921505] Lustre: lustre-MDT0000: new disk, initializing [ 9202.960117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9204.513852] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9207.923130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9208.087216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9209.726521] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9211.755285] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9213.231532] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9218.146679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9218.147417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9218.152061] Lustre: Skipped 1 previous similar message [ 9220.161649] LustreError: 400482:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9220.164502] LustreError: 400482:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9220.198162] Lustre: server umount lustre-MDT0000 complete [ 9221.768425] LustreError: 400485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353335 with bad export cookie 2308591593756112952 [ 9221.772094] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9221.773149] LustreError: 400485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9221.901441] Lustre: server umount lustre-MDT0001 complete [ 9225.728262] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 23:22:19 (1763353339) [ 9228.651512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9228.875543] LustreError: 401404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9228.886676] LustreError: 401404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9228.916608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9230.658547] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9234.392890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9234.400846] LustreError: 401404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9236.328198] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9238.867995] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9240.422726] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9243.396644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9246.295385] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9248.621104] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9262.300173] Lustre: Permanently deactivating lustre-OST0000 [ 9262.309613] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9268.723303] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9270.155361] LustreError: 404116:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 9270.160222] LustreError: 404116:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 9270.166068] LustreError: 404116:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9271.075066] LustreError: 404164:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 9271.079125] LustreError: 404164:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9277.938182] LustreError: 404261:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9277.942275] LustreError: 404261:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9277.997788] Lustre: server umount lustre-OST0000 complete [ 9279.107101] LustreError: 404413:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 9279.110371] LustreError: 404413:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9280.480480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9280.482209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9280.486135] Lustre: Skipped 1 previous similar message [ 9280.488389] Lustre: Skipped 1 previous similar message [ 9294.304503] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9294.360055] LustreError: 404510:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9294.362163] LustreError: 404510:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9294.409529] Lustre: server umount lustre-MDT0000 complete [ 9295.841074] LustreError: 401404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9295.848769] LustreError: 401404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9295.924437] LustreError: 403773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353409 with bad export cookie 2308591593756113491 [ 9295.928093] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9295.929249] LustreError: 403773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9306.150823] Lustre: server umount lustre-MDT0001 complete [ 9309.862842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9311.565682] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9312.687935] LustreError: 405555:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9315.402470] Lustre: 405893:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.127@tcp [ 9316.233967] LustreError: 405990:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9316.236379] LustreError: 405990:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9316.275429] Lustre: server umount MGS complete [ 9321.153387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9321.364437] LustreError: 406619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9321.375632] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9321.378609] Lustre: Skipped 1 previous similar message [ 9321.403662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9321.406805] Lustre: Skipped 2 previous similar messages [ 9323.089100] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9326.560030] LustreError: 406619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9326.603425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9326.730201] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9328.361458] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9330.570804] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9331.997447] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9332.749671] Lustre: Permanently reactivating lustre-OST0000 [ 9332.757859] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9332.760957] Lustre: Skipped 2 previous similar messages [ 9357.791141] LustreError: 408173:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 9360.880601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9361.035058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9361.038243] Lustre: Skipped 1 previous similar message [ 9363.374915] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9365.611523] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9378.784613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9378.786108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9378.786837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9378.786842] Lustre: Skipped 4 previous similar messages [ 9378.797155] Lustre: Skipped 1 previous similar message [ 9379.883196] LustreError: 409903:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9379.885582] LustreError: 409903:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 9379.927681] Lustre: server umount lustre-OST0000 complete [ 9383.392084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9383.600584] Lustre: server umount lustre-MDT0000 complete [ 9383.905066] LustreError: 408171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9385.154876] LustreError: 407967:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353498 with bad export cookie 2308591593756114436 [ 9385.158678] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9385.162543] LustreError: 407967:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9385.284099] Lustre: server umount lustre-MDT0001 complete [ 9389.493088] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 9390.683099] Key type lgssc unregistered [ 9390.832536] LNet: 410841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9390.836589] LNetError: 410841:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9390.849526] LNet: Removed LNI 192.168.204.127@tcp [ 9391.191814] Key type .llcrypt unregistered [ 9391.193283] Key type ._llcrypt unregistered [ 9398.712627] Key type ._llcrypt registered [ 9398.713873] Key type .llcrypt registered [ 9398.761762] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 9406.051244] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 9406.412315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9406.481879] alg: No test for adler32 (adler32-zlib) [ 9407.354317] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 9407.454724] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 9409.047187] Key type lgssc registered [ 9409.481110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9413.145408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9415.921484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9418.650450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9421.219191] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9423.873838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9423.897387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9423.903704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9425.002388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9425.016901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9425.056593] Lustre: lustre-MDT0000: new disk, initializing [ 9425.085582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9425.094243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9426.585657] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9431.193853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9431.217542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9431.256673] Lustre: 413945:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9431.281063] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9431.283670] Lustre: Skipped 1 previous similar message [ 9431.318462] Lustre: lustre-MDT0001: new disk, initializing [ 9431.354832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9431.370426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9431.374669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9432.857403] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9436.555954] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9438.027789] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9440.577044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9440.607717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9440.713429] Lustre: lustre-OST0000: new disk, initializing [ 9440.716531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9440.756115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9442.853235] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9446.382584] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9446.910094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9446.914340] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9446.957956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9448.041206] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9448.130519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9449.711042] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9449.806052] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9452.001150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9452.003634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9452.005359] LustreError: Skipped 1 previous similar message [ 9452.010760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9452.014752] Lustre: Skipped 1 previous similar message [ 9456.689076] LustreError: 416886:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9456.739612] Lustre: server umount lustre-OST0000 complete [ 9462.241580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9462.241629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9462.242325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9462.246168] Lustre: Skipped 2 previous similar messages [ 9464.322378] LustreError: 417087:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9464.324886] LustreError: 417087:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9464.358627] Lustre: server umount lustre-MDT0000 complete [ 9465.760188] LustreError: 413937:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353579 with bad export cookie 14299968257618059239 [ 9465.763618] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9465.764121] LustreError: 413937:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9465.828759] LustreError: 417289:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9465.830805] LustreError: 417289:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9465.883885] Lustre: server umount lustre-MDT0001 complete [ 9469.036822] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9469.697038] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 23:26:23 (1763353583) [ 9472.296082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9472.501468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9474.006848] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9477.336811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9477.513225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9479.010513] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9481.125104] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9482.613685] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9485.188299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9487.281222] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9489.282952] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9489.959577] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 9500.702185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 9502.690779] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9502.690892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9502.694382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9502.694394] Lustre: Skipped 1 previous similar message [ 9502.696076] Lustre: Skipped 1 previous similar message [ 9507.808365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9507.811110] Lustre: Skipped 2 previous similar messages [ 9508.397161] LustreError: 420465:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9508.399083] LustreError: 420465:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9508.438886] Lustre: server umount lustre-OST0000 complete [ 9512.929114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9512.937871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9512.940565] Lustre: Skipped 1 previous similar message [ 9523.168460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9523.171327] Lustre: Skipped 3 previous similar messages [ 9524.191158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9524.232859] LustreError: 420668:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9524.234844] LustreError: 420668:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9524.264546] Lustre: server umount lustre-MDT0000 complete [ 9525.755542] LustreError: 420669:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353639 with bad export cookie 14299968257618060121 [ 9525.759054] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9525.762704] LustreError: 420669:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9525.889110] Lustre: server umount lustre-MDT0001 complete [ 9530.012229] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 9531.189657] Key type lgssc unregistered [ 9531.346882] LNet: 421402:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9531.350512] LNetError: 421402:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9531.359936] LNet: Removed LNI 192.168.204.127@tcp [ 9531.707253] Key type .llcrypt unregistered [ 9531.708347] Key type ._llcrypt unregistered [ 9540.625821] Key type ._llcrypt registered [ 9540.627476] Key type .llcrypt registered [ 9540.673311] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 9540.807353] Key type .llcrypt unregistered [ 9540.808835] Key type ._llcrypt unregistered [ 9548.867657] Key type ._llcrypt registered [ 9548.869056] Key type .llcrypt registered [ 9548.915866] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 9549.297767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9549.343063] alg: No test for adler32 (adler32-zlib) [ 9550.221529] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 9550.320501] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 9551.903153] Key type lgssc registered [ 9552.332211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9555.293435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9555.304380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9556.514583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9558.070190] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9561.605104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9561.771696] LustreError: 423545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9561.777991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9561.780653] LustreError: 423545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9564.055647] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9566.102099] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9566.881564] LustreError: 423550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9566.887815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9566.889829] LustreError: 423550:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9568.707760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9568.875330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9570.446313] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9578.977151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9578.980770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9578.987317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9579.491500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9579.491657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9583.793122] LustreError: 425604:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 9583.834905] Lustre: server umount lustre-OST0000 complete [ 9588.194559] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 9588.197829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9588.203703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9588.206347] Lustre: Skipped 1 previous similar message [ 9591.485778] LustreError: 425806:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9591.488122] LustreError: 425806:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9591.521129] Lustre: server umount lustre-MDT0000 complete [ 9592.956363] LustreError: 423530:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353706 with bad export cookie 13413657282874066512 [ 9592.959757] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9592.961988] LustreError: 423530:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9593.027956] LustreError: 426007:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9593.030235] LustreError: 426007:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9593.098766] Lustre: server umount lustre-MDT0001 complete [ 9597.319178] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 9598.441767] Key type lgssc unregistered [ 9598.579393] LNet: 426539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9598.582883] LNetError: 426539:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9598.594373] LNet: Removed LNI 192.168.204.127@tcp [ 9598.925685] Key type .llcrypt unregistered [ 9598.927284] Key type ._llcrypt unregistered [ 9608.599483] Key type ._llcrypt registered [ 9608.602316] Key type .llcrypt registered [ 9608.653770] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 9608.998650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9609.064206] alg: No test for adler32 (adler32-zlib) [ 9609.927762] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 9610.017786] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 9611.599136] Key type lgssc registered [ 9611.990677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9614.841808] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9614.848961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9630.303295] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9635.423197] LustreError: 427886:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.127@tcp: failed processing log, type 4: rc = -110 [ 9662.047136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9663.972558] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9665.789624] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9668.114874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9668.331739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9668.356066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9669.720413] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9672.861606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9673.020492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9674.446124] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9676.411989] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9677.693768] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9682.401529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9682.403182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9682.404915] Lustre: Skipped 1 previous similar message [ 9686.189860] LustreError: 430165:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 9686.230638] Lustre: server umount lustre-OST0000 complete [ 9688.543799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9688.545622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9688.548466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9688.550027] Lustre: Skipped 1 previous similar message [ 9692.640092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9692.640579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9693.691729] LustreError: 430368:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 9693.694752] LustreError: 430368:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9693.721465] Lustre: server umount lustre-MDT0000 complete [ 9694.936769] LustreError: 427895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353808 with bad export cookie 16431499649856262919 [ 9694.939703] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9694.941062] LustreError: 427895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9695.008318] LustreError: 430571:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9695.010142] LustreError: 430571:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9695.064717] Lustre: server umount lustre-MDT0001 complete [ 9698.574823] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 9699.625403] Key type lgssc unregistered [ 9699.748366] LNet: 431104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9699.752452] LNetError: 431104:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9699.761288] LNet: Removed LNI 192.168.204.127@tcp [ 9700.036268] Key type .llcrypt unregistered [ 9700.037307] Key type ._llcrypt unregistered [ 9709.411947] Key type ._llcrypt registered [ 9709.413393] Key type .llcrypt registered [ 9709.458871] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 9709.787072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9709.794424] alg: No test for adler32 (adler32-zlib) [ 9710.644839] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 9710.736091] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 9712.327264] Key type lgssc registered [ 9712.732664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9715.502922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9715.510828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9716.704274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9718.133789] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9721.185697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9721.320076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9722.782836] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9724.690726] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9725.968285] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9728.491351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9728.629936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9730.748761] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9732.663069] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9742.959353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9746.911935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9746.914650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9746.919820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9748.432216] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.27@tcp (stopping) [ 9748.450197] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9750.273914] LustreError: 434746:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9750.302916] Lustre: server umount lustre-MDT0000 complete [ 9751.121973] LustreError: 432459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.27@tcp arrived at 1763353864 with bad export cookie 5490627984364787737 [ 9751.129153] LustreError: 432459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9751.803177] LustreError: 432459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353865 with bad export cookie 5490627984364787086 [ 9751.807409] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9751.807603] LustreError: 432459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9753.568094] LustreError: 432480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9753.568484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9753.576189] LustreError: 432480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9753.580183] Lustre: Skipped 1 previous similar message [ 9753.585656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9753.587076] Lustre: Skipped 3 previous similar messages [ 9756.575860] LustreError: 432480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9756.576104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9756.581229] LustreError: 432480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9757.872122] LustreError: 434949:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9757.875553] LustreError: 434949:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9757.920914] Lustre: server umount lustre-OST0000 complete [ 9760.309128] LustreError: 435303:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9760.311301] LustreError: 435303:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9760.368509] Lustre: server umount lustre-MDT0001 complete [ 9763.993327] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 9765.105547] Key type lgssc unregistered [ 9765.249457] LNet: 435834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9765.253379] LNetError: 435834:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9765.262421] LNet: Removed LNI 192.168.204.127@tcp [ 9765.584594] Key type .llcrypt unregistered [ 9765.586142] Key type ._llcrypt unregistered [ 9775.007876] Key type ._llcrypt registered [ 9775.009551] Key type .llcrypt registered [ 9775.051956] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 9775.508350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9775.521974] alg: No test for adler32 (adler32-zlib) [ 9776.395360] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 9776.490192] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 9778.079216] Key type lgssc registered [ 9778.496523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9781.592593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9781.601862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9782.784356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9784.276830] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9787.371677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9787.507699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9788.907962] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9790.742941] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9791.938545] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9794.480051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9794.601073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9796.421379] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9798.157936] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9807.906780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9812.961215] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9812.964954] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9812.966568] Lustre: Skipped 1 previous similar message [ 9813.967892] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.27@tcp (stopping) [ 9818.080476] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9818.083024] Lustre: Skipped 1 previous similar message [ 9823.200308] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9823.202143] Lustre: Skipped 2 previous similar messages [ 9824.223118] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9824.262202] LustreError: 439476:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9824.288969] Lustre: server umount lustre-MDT0001 complete [ 9828.320263] LustreError: 437209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9828.321104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9828.321209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9828.321218] Lustre: Skipped 1 previous similar message [ 9828.322143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9828.322147] Lustre: Skipped 1 previous similar message [ 9828.325829] LustreError: 437209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9832.110092] LustreError: 439681:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9832.113279] LustreError: 439681:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9832.156778] Lustre: server umount lustre-OST0000 complete [ 9833.440769] LustreError: 437209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9833.545745] LustreError: 439883:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9833.548014] LustreError: 439883:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9833.598191] Lustre: server umount lustre-MDT0000 complete [ 9838.312050] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 9839.344757] Key type lgssc unregistered [ 9839.474407] LNet: 440565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9839.477223] LNetError: 440565:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9839.483340] LNet: Removed LNI 192.168.204.127@tcp [ 9839.767440] Key type .llcrypt unregistered [ 9839.768491] Key type ._llcrypt unregistered [ 9848.076587] Key type ._llcrypt registered [ 9848.077467] Key type .llcrypt registered [ 9848.112242] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 9854.121451] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 9854.493732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9854.502418] alg: No test for adler32 (adler32-zlib) [ 9855.346051] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 9855.421724] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 9856.999178] Key type lgssc registered [ 9857.330560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9860.189531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9862.503940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9864.705144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9867.025882] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9870.444283] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 9874.216890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9876.647204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9876.667854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9876.671820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9877.749739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9877.761844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9877.800295] Lustre: lustre-MDT0000: new disk, initializing [ 9877.823272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9877.830601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9879.107321] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9882.594645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9884.945159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9884.971240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9885.064804] Lustre: lustre-OST0000: new disk, initializing [ 9885.066866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9885.068937] Lustre: Skipped 1 previous similar message [ 9885.094912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9885.693141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9885.697083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9885.705778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9887.045389] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9890.400624] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9893.773903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9896.367877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9896.399696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9896.448208] Lustre: 444512:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9896.488875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9896.530078] Lustre: lustre-MDT0001: new disk, initializing [ 9896.581249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9896.595462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 9896.600243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 9897.969993] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9901.803136] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9901.884414] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9903.297027] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9903.390247] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9905.141860] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.27@tcp (stopping) [ 9907.168459] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9907.170049] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9907.171157] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9907.174283] Lustre: Skipped 1 previous similar message [ 9907.176609] Lustre: Skipped 1 previous similar message [ 9910.216835] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.27@tcp (stopping) [ 9911.229720] LustreError: 447933:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9911.255373] Lustre: server umount lustre-MDT0001 complete [ 9917.409217] LustreError: 446738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9917.414936] LustreError: 446738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9922.529262] LustreError: 444519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9922.533666] LustreError: 444519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9926.623992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9926.627048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9926.631578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9927.649707] LustreError: 446738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9929.645072] LustreError: 448204:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 9929.647771] LustreError: 448204:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9929.697217] Lustre: server umount lustre-OST0000 complete [ 9937.149777] LustreError: 448405:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 9937.152373] LustreError: 448405:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9937.207885] Lustre: server umount lustre-MDT0000 complete [ 9941.670331] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 9942.664471] Key type lgssc unregistered [ 9942.790373] LNet: 449091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9942.792422] LNetError: 449091:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9942.798306] LNet: Removed LNI 192.168.204.127@tcp [ 9943.084342] Key type .llcrypt unregistered [ 9943.085691] Key type ._llcrypt unregistered [ 9955.806562] Key type ._llcrypt registered [ 9955.807569] Key type .llcrypt registered [ 9955.847701] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 9955.948276] Key type .llcrypt unregistered [ 9955.950139] Key type ._llcrypt unregistered [ 9962.698833] Key type ._llcrypt registered [ 9962.699947] Key type .llcrypt registered [ 9962.734621] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 9963.046991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9963.097051] alg: No test for adler32 (adler32-zlib) [ 9963.942618] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 9964.018217] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 9965.599183] Key type lgssc registered [ 9965.947197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9968.378854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9970.554218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9972.830735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9975.224861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9975.253377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9975.262431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9976.366473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9976.384258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9976.423225] Lustre: lustre-MDT0000: new disk, initializing [ 9976.453995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9976.464162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9977.861806] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9982.107591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9982.127192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9982.156372] Lustre: 452424:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9982.181072] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9982.182882] Lustre: Skipped 1 previous similar message [ 9982.219487] Lustre: lustre-MDT0001: new disk, initializing [ 9982.252409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9982.263163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9982.266385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9983.572244] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9986.766933] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9988.077864] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9990.622578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9990.646167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9990.744262] Lustre: lustre-OST0000: new disk, initializing [ 9990.746423] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9990.774265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9992.847231] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9996.083177] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9997.817342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9997.820487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9997.848933] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10008.034101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10008.034849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10008.035207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10008.037699] Lustre: Skipped 2 previous similar messages [10009.663184] LustreError: 454868:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10009.691656] Lustre: server umount lustre-MDT0000 complete [10011.048311] LustreError: 454168:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354124 with bad export cookie 10808953112026736435 [10011.052299] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10011.053675] LustreError: 454168:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10011.118216] LustreError: 455070:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10011.121316] LustreError: 455070:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10011.173419] Lustre: server umount lustre-MDT0001 complete [10022.768462] LustreError: 455271:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10022.770027] LustreError: 455271:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10022.809284] Lustre: server umount lustre-OST0000 complete [10026.169855] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 23:35:39 (1763354139) [10026.509496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10028.593671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10028.757339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10030.024749] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10032.744614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10032.863225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10034.149826] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10035.876772] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10037.008795] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10039.121033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10039.144596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10039.204286] Lustre: Found index 0 for lustre-OST0000, updating log [10041.089974] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10042.946839] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10043.372872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10053.601191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10053.603209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10053.605473] LustreError: Skipped 1 previous similar message [10053.610962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10053.615018] Lustre: Skipped 3 previous similar messages [10054.626264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10054.628195] Lustre: Skipped 1 previous similar message [10055.534071] LustreError: 458358:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10055.536381] LustreError: 458358:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10055.573104] Lustre: server umount lustre-OST0000 complete [10056.508810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10058.719858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10058.723391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10058.728547] Lustre: Skipped 1 previous similar message [10058.730722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10063.227179] LustreError: 458613:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10063.229463] LustreError: 458613:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10063.257428] Lustre: server umount lustre-MDT0000 complete [10064.521164] LustreError: 457717:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354178 with bad export cookie 10808953112026737814 [10064.524462] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10064.525318] LustreError: 457717:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10064.640509] Lustre: server umount lustre-MDT0001 complete [10067.421322] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 23:36:21 (1763354181) [10067.729066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10068.044177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10068.351577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10068.685642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10069.015188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10069.366738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10069.707182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10070.048037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10070.381823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10070.722277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10071.046506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10071.363902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10071.689605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10072.026784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10072.367662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10072.697273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10073.019345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10073.334286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10073.662190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10074.000637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10074.337355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10074.661361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10074.999038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10075.342721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10075.670257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10076.019687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10076.387917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10076.730575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10077.084054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10077.433466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10077.789391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10078.131421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10078.488487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10078.828628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10079.185949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10079.546383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10079.884589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10080.236559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10080.578966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10080.914527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10081.225118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10081.555189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10081.897725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10082.242426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10082.598335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10082.926877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10083.266652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10083.611923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10083.947978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10084.287757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10084.638114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10087.316081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10087.472409] LustreError: 462192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10087.480414] LustreError: 462192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10087.498578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10087.501475] Lustre: Skipped 1 previous similar message [10088.926754] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10092.108183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10093.583674] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10095.425382] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10096.722808] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10099.117696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10099.150929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10099.251740] Lustre: Found index 0 for lustre-OST0000, updating log [10100.191955] LNet: 1 local NIs in recovery (showing 1): 192.168.204.127@tcp [10100.397104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [10101.296597] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10103.105541] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10105.824535] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10105.827602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10105.832230] Lustre: Skipped 1 previous similar message [10105.834123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10105.836054] Lustre: Skipped 4 previous similar messages [10109.872182] LustreError: 464486:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10109.874787] LustreError: 464486:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10109.918617] Lustre: server umount lustre-OST0000 complete [10111.198754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10111.203040] Lustre: Skipped 2 previous similar messages [10114.272182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10114.274089] Lustre: Skipped 8 previous similar messages [10117.343089] Lustre: server umount lustre-MDT0000 complete [10117.346504] LustreError: 462192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10117.354144] LustreError: 462192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10118.369086] LustreError: 462192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10118.374812] LustreError: 462192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10118.703653] LustreError: 463855:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354232 with bad export cookie 10808953112026738885 [10118.707659] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10118.708272] LustreError: 463855:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10118.837185] Lustre: server umount lustre-MDT0001 complete [10119.844565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10125.252932] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [10128.145756] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [10131.158190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10133.186254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10135.111928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10137.073974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10139.097944] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:37:32 (1763354252) [10159.583449] LNet: 1 local NIs in recovery (showing 1): 192.168.204.127@tcp [10174.755529] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [10175.790718] Key type lgssc unregistered [10175.918640] LNet: 468482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10175.922632] LNetError: 468482:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10306.986204] LNet: Removed LNI 192.168.204.127@tcp [10307.285709] Key type .llcrypt unregistered [10307.287051] Key type ._llcrypt unregistered [10313.805563] Key type ._llcrypt registered [10313.807062] Key type .llcrypt registered [10313.843923] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [10314.140293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10314.148569] alg: No test for adler32 (adler32-zlib) [10315.003771] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [10315.088441] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [10316.671190] Key type lgssc registered [10317.047701] Lustre: Echo OBD driver; http://www.lustre.org/ [10318.534980] LNet: Added LNI 192.168.204.127@tcp100 [8/256/0/180] [10318.835401] LNet: Added LNI 192.168.204.127@tcp101 [8/256/0/180] [10319.141812] LNet: Added LNI 192.168.204.127@tcp102 [8/256/0/180] [10319.448965] LNet: Added LNI 192.168.204.127@tcp103 [8/256/0/180] [10319.738463] LNet: Added LNI 192.168.204.127@tcp104 [8/256/0/180] [10320.046125] LNet: Added LNI 192.168.204.127@tcp105 [8/256/0/180] [10320.339760] LNet: Added LNI 192.168.204.127@tcp106 [8/256/0/180] [10320.634470] LNet: Added LNI 192.168.204.127@tcp107 [8/256/0/180] [10320.919618] LNet: Added LNI 192.168.204.127@tcp108 [8/256/0/180] [10321.210970] LNet: Added LNI 192.168.204.127@tcp109 [8/256/0/180] [10321.513435] LNet: Added LNI 192.168.204.127@tcp110 [8/256/0/180] [10321.813614] LNet: Added LNI 192.168.204.127@tcp111 [8/256/0/180] [10322.096788] LNet: Added LNI 192.168.204.127@tcp112 [8/256/0/180] [10322.380245] LNet: Added LNI 192.168.204.127@tcp113 [8/256/0/180] [10322.657065] LNet: Added LNI 192.168.204.127@tcp114 [8/256/0/180] [10322.950322] LNet: Added LNI 192.168.204.127@tcp115 [8/256/0/180] [10323.253917] LNet: Added LNI 192.168.204.127@tcp116 [8/256/0/180] [10323.549969] LNet: Added LNI 192.168.204.127@tcp117 [8/256/0/180] [10323.846211] LNet: Added LNI 192.168.204.127@tcp118 [8/256/0/180] [10324.131164] LNet: Added LNI 192.168.204.127@tcp119 [8/256/0/180] [10324.426039] LNet: Added LNI 192.168.204.127@tcp120 [8/256/0/180] [10324.719663] LNet: Added LNI 192.168.204.127@tcp121 [8/256/0/180] [10325.017445] LNet: Added LNI 192.168.204.127@tcp122 [8/256/0/180] [10325.318038] LNet: Added LNI 192.168.204.127@tcp123 [8/256/0/180] [10325.615775] LNet: Added LNI 192.168.204.127@tcp124 [8/256/0/180] [10325.910081] LNet: Added LNI 192.168.204.127@tcp125 [8/256/0/180] [10326.208757] LNet: Added LNI 192.168.204.127@tcp126 [8/256/0/180] [10326.496382] LNet: Added LNI 192.168.204.127@tcp127 [8/256/0/180] [10326.773840] LNet: Added LNI 192.168.204.127@tcp128 [8/256/0/180] [10327.046066] LNet: Added LNI 192.168.204.127@tcp129 [8/256/0/180] [10327.350307] LNet: Added LNI 192.168.204.127@tcp130 [8/256/0/180] [10327.648131] LNet: Added LNI 192.168.204.127@tcp131 [8/256/0/180] [10327.939672] LNet: Added LNI 192.168.204.127@tcp132 [8/256/0/180] [10328.227875] LNet: Added LNI 192.168.204.127@tcp133 [8/256/0/180] [10328.513705] LNet: Added LNI 192.168.204.127@tcp134 [8/256/0/180] [10328.797653] LNet: Added LNI 192.168.204.127@tcp135 [8/256/0/180] [10330.986357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10331.008497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10331.012819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10332.053205] Lustre: 471472:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.127@tcp: can't add NID 192.168.204.127@tcp130: rc = -75 [10332.081410] LNetError: 471494:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [10332.085676] LNetError: Protocol error connecting to 192.168.204.127@tcp122 at host 192.168.204.127:988: is it running a compatible version of Lustre? [10332.090853] LNetError: Skipped 4 previous similar messages [10332.133595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10332.144495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10332.181199] Lustre: lustre-MDT0000: new disk, initializing [10332.211047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10332.219393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10332.511102] LNet: 2 local NIs in recovery (showing 2): 192.168.204.127@tcp125, 192.168.204.127@tcp122 [10333.153745] LNetError: 471504:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [10333.156335] LNetError: 471504:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 4 previous similar messages [10333.160058] LNetError: Protocol error connecting to 192.168.204.127@tcp107 at host 192.168.204.127:988: is it running a compatible version of Lustre? [10333.494953] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10335.584399] LNetError: 471493:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [10335.589279] LNetError: 471493:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 2 previous similar messages [10335.592054] LNetError: Protocol error connecting to 192.168.204.127@tcp125 at host 192.168.204.127:988: is it running a compatible version of Lustre? [10335.594685] LNetError: Skipped 2 previous similar messages [10337.641967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10337.665221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10337.709262] Lustre: 471491:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10337.736423] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10337.739189] Lustre: Skipped 1 previous similar message [10337.784169] Lustre: lustre-MDT0001: new disk, initializing [10337.796723] Lustre: 472279:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.127@tcp: can't add NID 192.168.204.127@tcp130: rc = -75 [10337.801374] Lustre: 472279:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [10337.864636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10337.877943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10337.881233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10339.109512] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10339.680427] LNetError: 468936:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [10339.685026] LNetError: Protocol error connecting to 192.168.204.127@tcp125 at host 192.168.204.127:988: is it running a compatible version of Lustre? [10342.093898] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10343.212597] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10345.352722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10345.374397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10345.472944] Lustre: lustre-OST0000: new disk, initializing [10345.475631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10345.511087] Lustre: 473249:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.127@tcp: can't add NID 192.168.204.127@tcp130: rc = -75 [10345.513150] Lustre: 473249:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [10345.527671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10347.357677] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10347.871878] LNetError: 468934:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [10347.874800] LNetError: Protocol error connecting to 192.168.204.127@tcp125 at host 192.168.204.127:988: is it running a compatible version of Lustre? [10350.347222] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10350.577868] Lustre: 473845:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.127@tcp: can't add NID 192.168.204.127@tcp130: rc = -75 [10350.580614] Lustre: 473845:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [10350.594899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10350.598479] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10350.632702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10355.681627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10355.682661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10355.683388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10355.688870] Lustre: Skipped 1 previous similar message [10358.766106] LustreError: 474085:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10358.803689] Lustre: server umount lustre-OST0000 complete [10360.800303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10360.800978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10360.803743] Lustre: Skipped 1 previous similar message [10360.806094] Lustre: Skipped 1 previous similar message [10363.231968] LNetError: 471498:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [10363.234856] LNetError: Protocol error connecting to 192.168.204.127@tcp125 at host 192.168.204.127:988: is it running a compatible version of Lustre? [10365.920235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10365.922964] Lustre: Skipped 2 previous similar messages [10366.138650] LustreError: 474287:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10366.140388] LustreError: 474287:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10366.163524] Lustre: server umount lustre-MDT0000 complete [10367.376296] LustreError: 471482:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354480 with bad export cookie 11451967314295497877 [10367.379095] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10367.379554] LustreError: 471482:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10367.448531] LustreError: 474488:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10367.450610] LustreError: 474488:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10367.514634] Lustre: server umount lustre-MDT0001 complete [10370.951179] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [10372.011609] Key type lgssc unregistered [10372.166019] LNet: 475027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10372.168933] LNetError: 475027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10372.172276] LNet: Removed LNI 192.168.204.127@tcp [10373.215188] LNet: Removed LNI 192.168.204.127@tcp100 [10373.216347] LNetError: 475027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10373.217941] LNetError: 475027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10373.219845] LNet: Removed LNI 192.168.204.127@tcp101 [10374.239198] LNet: Removed LNI 192.168.204.127@tcp102 [10374.240944] LNetError: 475027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10374.243361] LNetError: 475027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10374.246575] LNet: Removed LNI 192.168.204.127@tcp103 [10375.263183] LNet: Removed LNI 192.168.204.127@tcp104 [10376.287183] LNet: Removed LNI 192.168.204.127@tcp105 [10376.288434] LNetError: 475027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10376.290490] LNetError: 475027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [10376.292816] LNet: Removed LNI 192.168.204.127@tcp106 [10377.311092] LNet: Removed LNI 192.168.204.127@tcp107 [10378.335140] LNet: Removed LNI 192.168.204.127@tcp108 [10379.359147] LNet: Removed LNI 192.168.204.127@tcp109 [10380.383127] LNet: Removed LNI 192.168.204.127@tcp110 [10380.384733] LNetError: 475027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10380.387542] LNetError: 475027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [10380.390485] LNet: Removed LNI 192.168.204.127@tcp111 [10381.407129] LNet: Removed LNI 192.168.204.127@tcp112 [10382.431148] LNet: Removed LNI 192.168.204.127@tcp113 [10383.455114] LNet: Removed LNI 192.168.204.127@tcp114 [10384.479132] LNet: Removed LNI 192.168.204.127@tcp115 [10385.503124] LNet: Removed LNI 192.168.204.127@tcp116 [10386.527150] LNet: Removed LNI 192.168.204.127@tcp117 [10387.551130] LNet: Removed LNI 192.168.204.127@tcp118 [10388.575176] LNet: Removed LNI 192.168.204.127@tcp119 [10388.577121] LNetError: 475027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10388.579935] LNetError: 475027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [10388.583440] LNet: Removed LNI 192.168.204.127@tcp120 [10389.599143] LNet: Removed LNI 192.168.204.127@tcp121 [10390.623123] LNet: Removed LNI 192.168.204.127@tcp122 [10391.647120] LNet: Removed LNI 192.168.204.127@tcp123 [10392.671138] LNet: Removed LNI 192.168.204.127@tcp124 [10392.673286] LNet: Removed LNI 192.168.204.127@tcp125 [10392.675353] LNet: Removed LNI 192.168.204.127@tcp126 [10392.677342] LNet: Removed LNI 192.168.204.127@tcp127 [10392.679354] LNet: Removed LNI 192.168.204.127@tcp128 [10392.680782] LNet: Removed LNI 192.168.204.127@tcp129 [10392.682156] LNet: Removed LNI 192.168.204.127@tcp130 [10392.683447] LNet: Removed LNI 192.168.204.127@tcp131 [10392.684879] LNet: Removed LNI 192.168.204.127@tcp132 [10392.686722] LNet: Removed LNI 192.168.204.127@tcp133 [10392.688603] LNet: Removed LNI 192.168.204.127@tcp134 [10392.696493] LNet: Removed LNI 192.168.204.127@tcp135 [10392.958835] Key type .llcrypt unregistered [10392.960578] Key type ._llcrypt unregistered [10400.213551] Key type ._llcrypt registered [10400.214408] Key type .llcrypt registered [10400.249116] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [10406.003603] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [10406.426776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10406.433537] alg: No test for adler32 (adler32-zlib) [10407.290229] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [10407.365673] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [10408.943252] Key type lgssc registered [10409.304916] Lustre: Echo OBD driver; http://www.lustre.org/ [10412.264275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10414.395347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10416.403962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10418.564550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10420.729650] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:42:14 (1763354534) [10426.349519] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [10427.278924] Key type lgssc unregistered [10427.401470] LNet: 479038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10427.403556] LNetError: 479038:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10427.411466] LNet: Removed LNI 192.168.204.127@tcp [10427.724515] Key type .llcrypt unregistered [10427.725397] Key type ._llcrypt unregistered [10434.150716] Key type ._llcrypt registered [10434.151653] Key type .llcrypt registered [10434.185325] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [10434.536721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10434.550503] alg: No test for adler32 (adler32-zlib) [10435.395615] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [10435.472368] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [10437.047147] Key type lgssc registered [10437.397622] Lustre: Echo OBD driver; http://www.lustre.org/ [10440.302882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10440.322477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10440.327660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10441.408180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10441.419916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10441.458364] Lustre: lustre-MDT0000: new disk, initializing [10441.484463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10441.492774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10442.747472] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10446.772575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10446.796820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10446.829788] Lustre: 480209:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10446.850278] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10446.852000] Lustre: Skipped 1 previous similar message [10446.889173] Lustre: lustre-MDT0001: new disk, initializing [10446.921386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10446.931570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10446.935267] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10448.146556] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10451.070223] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10452.172020] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10454.342787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10454.370169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10454.475977] Lustre: lustre-OST0000: new disk, initializing [10454.477721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10454.507340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10456.311525] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10459.334885] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10459.637598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10459.641379] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10459.666091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10460.417279] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing lnet_if_list [10461.976724] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [10461.987672] LNet: Added LNI 192.168.204.127@tcp10 [8/256/0/180] [10463.594937] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [10463.604957] LNet: Added LNI 192.168.204.127@tcp11 [8/256/0/180] [10464.807941] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [10464.821697] LNet: Added LNI 192.168.204.127@tcp12 [8/256/0/180] [10466.057563] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [10466.068862] LNet: Added LNI 192.168.204.127@tcp13 [8/256/0/180] [10467.367967] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [10467.380283] LNet: Added LNI 192.168.204.127@tcp14 [8/256/0/180] [10468.698367] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [10468.707686] LNet: Added LNI 192.168.204.127@tcp15 [8/256/0/180] [10470.105708] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [10470.116151] LNet: Added LNI 192.168.204.127@tcp16 [8/256/0/180] [10471.583778] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [10471.594691] LNet: Added LNI 192.168.204.127@tcp17 [8/256/0/180] [10473.128153] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [10473.138422] LNet: Added LNI 192.168.204.127@tcp18 [8/256/0/180] [10474.678975] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [10474.691815] LNet: Added LNI 192.168.204.127@tcp19 [8/256/0/180] [10476.197465] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [10476.207883] LNet: Added LNI 192.168.204.127@tcp20 [8/256/0/180] [10477.743096] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [10477.753536] LNet: Added LNI 192.168.204.127@tcp21 [8/256/0/180] [10479.370272] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [10479.381477] LNet: Added LNI 192.168.204.127@tcp22 [8/256/0/180] [10480.911333] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [10480.924453] LNet: Added LNI 192.168.204.127@tcp23 [8/256/0/180] [10482.521331] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [10482.529610] LNet: Added LNI 192.168.204.127@tcp24 [8/256/0/180] [10484.158578] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [10484.167398] LNet: Added LNI 192.168.204.127@tcp25 [8/256/0/180] [10485.763649] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [10485.773772] LNet: Added LNI 192.168.204.127@tcp26 [8/256/0/180] [10487.560042] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [10487.570688] LNet: Added LNI 192.168.204.127@tcp27 [8/256/0/180] [10487.577573] LNetError: 479471:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [10487.581482] LNetError: Protocol error connecting to 192.168.204.127@tcp26 at host 192.168.204.127:988: is it running a compatible version of Lustre? [10489.335593] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [10489.346657] LNet: Added LNI 192.168.204.127@tcp28 [8/256/0/180] [10489.350687] LNetError: 479470:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [10489.355905] LNetError: Protocol error connecting to 192.168.204.127@tcp26 at host 192.168.204.127:988: is it running a compatible version of Lustre? [10491.131801] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [10491.140744] LNet: Added LNI 192.168.204.127@tcp29 [8/256/0/180] [10491.143957] LNetError: 479469:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [10491.148355] LNetError: Protocol error connecting to 192.168.204.127@tcp28 at host 192.168.204.127:988: is it running a compatible version of Lustre? [10491.807117] LNet: 2 local NIs in recovery (showing 2): 192.168.204.127@tcp28, 192.168.204.127@tcp29 [10493.126793] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [10493.140585] LNet: Added LNI 192.168.204.127@tcp30 [8/256/0/180] [10494.881605] LNetError: 479471:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [10494.884508] LNetError: 479471:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 3 previous similar messages [10494.887601] LNetError: Protocol error connecting to 192.168.204.127@tcp29 at host 192.168.204.127:988: is it running a compatible version of Lustre? [10494.890283] LNetError: Skipped 3 previous similar messages [10495.257747] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [10495.268294] LNet: Added LNI 192.168.204.127@tcp31 [8/256/0/180] [10497.369928] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [10497.382125] LNet: Added LNI 192.168.204.127@tcp32 [8/256/0/180] [10498.977622] LNetError: 479469:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [10498.981294] LNetError: 479469:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 2 previous similar messages [10498.983762] LNetError: Protocol error connecting to 192.168.204.127@tcp30 at host 192.168.204.127:988: is it running a compatible version of Lustre? [10498.988315] LNetError: Skipped 2 previous similar messages [10499.667697] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [10499.678689] LNet: Added LNI 192.168.204.127@tcp33 [8/256/0/180] [10502.000491] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [10502.012695] LNet: Added LNI 192.168.204.127@tcp34 [8/256/0/180] [10504.376660] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [10504.387262] LNet: Added LNI 192.168.204.127@tcp35 [8/256/0/180] [10506.788267] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [10506.802909] LNet: Added LNI 192.168.204.127@tcp36 [8/256/0/180] [10507.169465] LNetError: 479468:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [10507.174187] LNetError: 479468:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 7 previous similar messages [10507.176731] LNetError: Protocol error connecting to 192.168.204.127@tcp31 at host 192.168.204.127:988: is it running a compatible version of Lustre? [10507.181224] LNetError: Skipped 7 previous similar messages [10509.205267] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [10509.216176] LNet: Added LNI 192.168.204.127@tcp37 [8/256/0/180] [10511.600735] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [10511.611132] LNet: Added LNI 192.168.204.127@tcp38 [8/256/0/180] [10513.917582] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [10513.929895] LNet: Added LNI 192.168.204.127@tcp39 [8/256/0/180] [10516.318186] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [10516.329818] LNet: Added LNI 192.168.204.127@tcp40 [8/256/0/180] [10518.900502] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [10518.913292] LNet: Added LNI 192.168.204.127@tcp41 [8/256/0/180] [10521.460569] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [10521.472120] LNet: Added LNI 192.168.204.127@tcp42 [8/256/0/180] [10523.552166] LNetError: 479469:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [10523.555371] LNetError: 479469:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 20 previous similar messages [10523.557886] LNetError: Protocol error connecting to 192.168.204.127@tcp26 at host 192.168.204.127:988: is it running a compatible version of Lustre? [10523.560576] LNetError: Skipped 20 previous similar messages [10523.854784] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [10523.865404] LNet: Added LNI 192.168.204.127@tcp43 [8/256/0/180] [10526.252779] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [10526.263921] LNet: Added LNI 192.168.204.127@tcp44 [8/256/0/180] [10528.786342] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [10528.797781] LNet: Added LNI 192.168.204.127@tcp45 [8/256/0/180] [10531.280534] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [10531.292174] LNet: Added LNI 192.168.204.127@tcp46 [8/256/0/180] [10534.028498] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [10534.042136] LNet: Added LNI 192.168.204.127@tcp47 [8/256/0/180] [10536.482332] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [10536.492614] LNet: Added LNI 192.168.204.127@tcp48 [8/256/0/180] [10538.834526] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [10538.845398] LNet: Added LNI 192.168.204.127@tcp49 [8/256/0/180] [10541.581694] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [10541.593981] LNet: Added LNI 192.168.204.127@tcp50 [8/256/0/180] [10551.779179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10551.780063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10551.781294] LustreError: Skipped 1 previous similar message [10551.783171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10551.785414] Lustre: Skipped 1 previous similar message [10552.223201] LNet: 5 local NIs in recovery (showing 5): 192.168.204.127@tcp44, 192.168.204.127@tcp47, 192.168.204.127@tcp41, 192.168.204.127@tcp40, 192.168.204.127@tcp30 [10555.437077] LustreError: 504388:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10555.474869] Lustre: server umount lustre-OST0000 complete [10556.896308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10556.896928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10556.900414] Lustre: Skipped 1 previous similar message [10556.901669] Lustre: Skipped 1 previous similar message [10557.345092] LNetError: 479471:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.127 [10557.348944] LNetError: 479471:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 50 previous similar messages [10557.351120] LNetError: Protocol error connecting to 192.168.204.127@tcp41 at host 192.168.204.127:988: is it running a compatible version of Lustre? [10557.353469] LNetError: Skipped 50 previous similar messages [10562.015972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10562.018174] Lustre: Skipped 2 previous similar messages [10567.136074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10567.139307] Lustre: Skipped 1 previous similar message [10571.231104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10571.267554] LustreError: 504589:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10571.269452] LustreError: 504589:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10571.303202] Lustre: server umount lustre-MDT0000 complete [10572.533223] LustreError: 480199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354686 with bad export cookie 18253868944467522629 [10572.536311] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10572.537730] LustreError: 480199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10572.601230] LustreError: 504793:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10572.603979] LustreError: 504793:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10572.660109] Lustre: server umount lustre-MDT0001 complete [10576.046308] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [10577.085879] Key type lgssc unregistered [10577.258063] LNet: 505331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10577.261952] LNetError: 505331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10577.264951] LNet: Removed LNI 192.168.204.127@tcp [10578.271158] LNet: Removed LNI 192.168.204.127@tcp10 [10578.272814] LNetError: 505331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10578.274630] LNetError: 505331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10578.276930] LNet: Removed LNI 192.168.204.127@tcp11 [10579.295173] LNet: Removed LNI 192.168.204.127@tcp12 [10579.296922] LNetError: 505331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10579.299947] LNetError: 505331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10579.303209] LNet: Removed LNI 192.168.204.127@tcp13 [10580.319121] LNet: Removed LNI 192.168.204.127@tcp14 [10581.343138] LNet: Removed LNI 192.168.204.127@tcp15 [10581.344323] LNetError: 505331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10581.347217] LNetError: 505331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [10581.350527] LNet: Removed LNI 192.168.204.127@tcp16 [10582.367149] LNet: Removed LNI 192.168.204.127@tcp17 [10583.391159] LNet: Removed LNI 192.168.204.127@tcp18 [10584.415134] LNet: Removed LNI 192.168.204.127@tcp19 [10585.439157] LNet: Removed LNI 192.168.204.127@tcp20 [10585.440682] LNetError: 505331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10585.443576] LNetError: 505331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [10585.445703] LNet: Removed LNI 192.168.204.127@tcp21 [10586.463237] LNet: Removed LNI 192.168.204.127@tcp22 [10587.487215] LNet: Removed LNI 192.168.204.127@tcp23 [10588.511161] LNet: Removed LNI 192.168.204.127@tcp24 [10589.535134] LNet: Removed LNI 192.168.204.127@tcp25 [10590.559150] LNet: Removed LNI 192.168.204.127@tcp26 [10591.583150] LNet: Removed LNI 192.168.204.127@tcp27 [10592.607160] LNet: Removed LNI 192.168.204.127@tcp28 [10593.631189] LNet: Removed LNI 192.168.204.127@tcp29 [10593.632324] LNetError: 505331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10593.634038] LNetError: 505331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [10593.636103] LNet: Removed LNI 192.168.204.127@tcp30 [10594.655207] LNet: Removed LNI 192.168.204.127@tcp31 [10595.679184] LNet: Removed LNI 192.168.204.127@tcp32 [10596.703167] LNet: Removed LNI 192.168.204.127@tcp33 [10597.727177] LNet: Removed LNI 192.168.204.127@tcp34 [10598.751132] LNet: Removed LNI 192.168.204.127@tcp35 [10599.775424] LNet: Removed LNI 192.168.204.127@tcp36 [10600.799184] LNet: Removed LNI 192.168.204.127@tcp37 [10601.823153] LNet: Removed LNI 192.168.204.127@tcp38 [10602.847168] LNet: Removed LNI 192.168.204.127@tcp39 [10602.848582] LNet: Removed LNI 192.168.204.127@tcp40 [10602.849827] LNet: Removed LNI 192.168.204.127@tcp41 [10603.871222] LNet: Removed LNI 192.168.204.127@tcp42 [10604.895164] LNet: Removed LNI 192.168.204.127@tcp43 [10604.896566] LNet: Removed LNI 192.168.204.127@tcp44 [10605.919125] LNet: Removed LNI 192.168.204.127@tcp45 [10606.943124] LNet: Removed LNI 192.168.204.127@tcp46 [10607.967165] LNet: Removed LNI 192.168.204.127@tcp47 [10608.991151] LNet: Removed LNI 192.168.204.127@tcp48 [10610.015184] LNet: Removed LNI 192.168.204.127@tcp49 [10610.016693] LNetError: 505331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10610.018413] LNetError: 505331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 19 previous similar messages [10610.026506] LNet: Removed LNI 192.168.204.127@tcp50 [10610.322909] Key type .llcrypt unregistered [10610.324066] Key type ._llcrypt unregistered [10617.797180] Key type ._llcrypt registered [10617.798029] Key type .llcrypt registered [10617.833821] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [10618.145340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10618.157865] alg: No test for adler32 (adler32-zlib) [10619.000414] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [10619.071955] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [10620.647095] Key type lgssc registered [10621.015738] Lustre: Echo OBD driver; http://www.lustre.org/ [10623.519386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10623.525283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10624.660832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10625.851097] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10628.613790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10628.735959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10629.991402] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10631.719903] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10632.809543] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10634.933751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10635.041818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10636.899762] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10638.583652] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10643.043612] loop0: detected capacity change from 0 to 204800000 [10643.113362] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10643.116291] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10643.119970] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10643.145455] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10645.026707] loop0: detected capacity change from 0 to 204800000 [10645.075486] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10645.079892] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10645.085026] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10645.131201] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10646.763873] loop0: detected capacity change from 0 to 204800000 [10648.602853] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10648.623458] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10648.665700] Lustre: fs15246-OST0000: new disk, initializing [10648.667505] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [10648.684626] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10650.506693] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10654.179302] loop1: detected capacity change from 0 to 204800000 [10656.016703] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10656.045914] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10656.075380] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [10656.100496] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [10656.132347] Lustre: fs15246-MDT0000: new disk, initializing [10656.174903] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10656.182641] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10656.214177] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10657.247723] Lustre: 510927:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000c5b4a20e x1849011451490560/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763354786 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [10657.401101] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10662.304686] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10662.306900] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10662.311518] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10666.979629] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10666.985249] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10670.907073] LustreError: 512054:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [10670.928086] Lustre: server umount fs15246-MDT0000 complete [10672.099146] LustreError: 507268:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10672.104415] LustreError: 507268:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10673.904101] LustreError: 512556:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [10673.905668] LustreError: 512556:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10673.920139] Lustre: server umount fs15246-OST0000 complete [10677.217562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10677.217956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10677.226077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10677.227949] Lustre: Skipped 1 previous similar message [10682.338402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10682.340237] Lustre: Skipped 1 previous similar message [10683.182123] LustreError: 513056:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10683.185348] LustreError: 513056:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10683.227573] Lustre: server umount lustre-OST0000 complete [10685.407924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10685.412101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10685.415884] Lustre: Skipped 1 previous similar message [10685.417651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10687.457293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10690.624285] LustreError: 513258:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10690.626261] LustreError: 513258:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10690.652328] Lustre: server umount lustre-MDT0000 complete [10691.970438] LustreError: 506538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354805 with bad export cookie 10217358537476726961 [10691.974046] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10691.977043] LustreError: 506538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10692.094397] Lustre: server umount lustre-MDT0001 complete [10695.978269] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [10697.068044] Key type lgssc unregistered [10697.207367] LNet: 513991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10697.209669] LNetError: 513991:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10697.218372] LNet: Removed LNI 192.168.204.127@tcp [10697.539855] Key type .llcrypt unregistered [10697.541269] Key type ._llcrypt unregistered [10705.117168] Key type ._llcrypt registered [10705.118279] Key type .llcrypt registered [10705.158939] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [10705.493548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10705.519091] alg: No test for adler32 (adler32-zlib) [10706.380460] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [10706.474305] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [10708.063165] Key type lgssc registered [10708.435896] Lustre: Echo OBD driver; http://www.lustre.org/ [10710.791275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10712.362310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10714.073199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10715.591624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10720.971571] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [10723.997481] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [10727.156016] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10729.235331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10731.469538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10733.729315] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10735.961268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10735.978399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10735.982893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10737.068391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10737.078953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10737.119309] Lustre: lustre-MDT0000: new disk, initializing [10737.142642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10737.148432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10738.461066] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10742.541226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10742.565985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10742.601139] Lustre: 518393:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10742.634670] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10742.637282] Lustre: Skipped 1 previous similar message [10742.673108] Lustre: lustre-MDT0001: new disk, initializing [10742.706487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10742.719000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10742.721638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10744.149985] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10747.379263] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10748.517071] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10750.707977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10750.731857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10750.832836] Lustre: lustre-OST0000: new disk, initializing [10750.837868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10750.864418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10752.704150] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10754.042537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10754.046560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10754.076108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10755.865402] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10757.175509] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10757.242182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10758.519616] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10758.598603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10761.261104] LustreError: 521332:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10761.306924] Lustre: server umount lustre-OST0000 complete [10763.231860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10763.235110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10763.240922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10764.256934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10764.257398] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10769.376357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10769.379325] Lustre: Skipped 2 previous similar messages [10772.451757] LustreError: 521533:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10772.454186] LustreError: 521533:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10772.481198] Lustre: server umount lustre-MDT0000 complete [10773.778389] LustreError: 520139:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354887 with bad export cookie 12801530976294896567 [10773.782656] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10773.783825] LustreError: 520139:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10784.227856] LustreError: 521736:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10784.229937] LustreError: 521736:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10784.288509] Lustre: server umount lustre-MDT0001 complete [10793.063062] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [10796.631564] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [10800.350935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10802.906824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10805.144662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10807.474614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10808.079135] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10569 sec ========= 23:48:41 (1763354921) [10808.651708] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:48:42 (1763354922) === [10809.900388] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:48:43 (1763354923) === [10823.631715] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [10824.823458] Key type lgssc unregistered [10824.966447] LNet: 526584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10824.970489] LNetError: 526584:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10824.978320] LNet: Removed LNI 192.168.204.127@tcp [10825.308930] Key type .llcrypt unregistered [10825.310798] Key type ._llcrypt unregistered