[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 485913008 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2868616K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002334] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007555] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010128] LSM: Security Framework initializing [ 0.011066] Yama: becoming mindful. [ 0.012047] SELinux: Initializing. [ 0.013000] *** VALIDATE selinux *** [ 0.019742] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024835] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026171] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028117] *** VALIDATE tmpfs *** [ 0.029449] *** VALIDATE proc *** [ 0.030240] *** VALIDATE cgroup *** [ 0.031000] *** VALIDATE cgroup2 *** [ 0.031280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034029] Spectre V2 : User space: Vulnerable [ 0.035008] Speculative Store Bypass: Vulnerable [ 0.038619] debug: unmapping init [mem 0xffffffff97e59000-0xffffffff97e60fff] [ 0.040164] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041817] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042019] ... version: 2 [ 0.043012] ... bit width: 48 [ 0.044012] ... generic registers: 4 [ 0.045013] ... value mask: 0000ffffffffffff [ 0.046014] ... max period: 00007fffffffffff [ 0.047012] ... fixed-purpose events: 3 [ 0.048011] ... event mask: 000000070000000f [ 0.049299] rcu: Hierarchical SRCU implementation. [ 0.051429] smp: Bringing up secondary CPUs ... [ 0.052533] x86: Booting SMP configuration: [ 0.053025] .... node #0, CPUs: #1 #2 #3 [ 0.056483] smp: Brought up 1 node, 4 CPUs [ 0.058011] smpboot: Max logical packages: 1 [ 0.059016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137084] node 0 deferred pages initialised in 75ms [ 0.140052] devtmpfs: initialized [ 0.141408] x86/mm: Memory block size: 128MB [ 0.144842] gcov: version magic: 0x41383552 [ 0.147322] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149262] pinctrl core: initialized pinctrl subsystem [ 0.150181] [ 0.150587] ************************************************************* [ 0.151015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152012] ** ** [ 0.153016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154012] ** ** [ 0.155012] ** This means that this kernel is built to expose internal ** [ 0.156012] ** IOMMU data structures, which may compromise security on ** [ 0.157011] ** your system. ** [ 0.158013] ** ** [ 0.159013] ** If you see this message and you are not debugging the ** [ 0.160012] ** kernel, report this immediately to your vendor! ** [ 0.161013] ** ** [ 0.162014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163015] ************************************************************* [ 0.164796] NET: Registered protocol family 16 [ 0.166509] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.170067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173036] cpuidle: using governor menu [ 0.175792] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178443] PCI: Using configuration type 1 for base access [ 0.180128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189106] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.191031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.194223] cryptd: max_cpu_qlen set to 1000 [ 0.197218] ACPI: Added _OSI(Module Device) [ 0.199012] ACPI: Added _OSI(Processor Device) [ 0.201014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.202012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.206546] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.213111] ACPI: Interpreter enabled [ 0.214065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.215014] ACPI: Using IOAPIC for interrupt routing [ 0.217097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.221459] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.231778] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.235048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.236015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.238080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.243224] acpiphp: Slot [2] registered [ 0.245098] acpiphp: Slot [3] registered [ 0.246132] acpiphp: Slot [4] registered [ 0.247087] acpiphp: Slot [5] registered [ 0.249082] acpiphp: Slot [6] registered [ 0.250110] acpiphp: Slot [7] registered [ 0.251090] acpiphp: Slot [8] registered [ 0.253104] acpiphp: Slot [9] registered [ 0.254101] acpiphp: Slot [10] registered [ 0.256149] acpiphp: Slot [11] registered [ 0.257113] acpiphp: Slot [12] registered [ 0.259103] acpiphp: Slot [13] registered [ 0.260105] acpiphp: Slot [14] registered [ 0.262122] acpiphp: Slot [15] registered [ 0.263101] acpiphp: Slot [16] registered [ 0.265110] acpiphp: Slot [17] registered [ 0.267122] acpiphp: Slot [18] registered [ 0.268125] acpiphp: Slot [19] registered [ 0.270099] acpiphp: Slot [20] registered [ 0.271095] acpiphp: Slot [21] registered [ 0.273108] acpiphp: Slot [22] registered [ 0.274099] acpiphp: Slot [23] registered [ 0.276131] acpiphp: Slot [24] registered [ 0.277102] acpiphp: Slot [25] registered [ 0.279098] acpiphp: Slot [26] registered [ 0.280109] acpiphp: Slot [27] registered [ 0.281101] acpiphp: Slot [28] registered [ 0.283091] acpiphp: Slot [29] registered [ 0.284114] acpiphp: Slot [30] registered [ 0.286091] acpiphp: Slot [31] registered [ 0.287058] PCI host bridge to bus 0000:00 [ 0.289022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.291038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.294025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.296016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.298025] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.301028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.303178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.307541] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.312325] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.321016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.326686] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.329018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.331020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.336418] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.339014] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.342042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344730] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.349013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.363016] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.369012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.372260] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.380016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.384016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.396016] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.403316] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.411015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.420018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.434022] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.445665] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.450015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.456018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.468015] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.478675] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.485017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.492025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.507016] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.518150] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.524017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.533019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.547030] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.559544] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.566015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.576016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.589016] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.599256] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.602196] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.605347] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.608419] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.610247] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.615289] iommu: Default domain type: Passthrough [ 0.617433] SCSI subsystem initialized [ 0.619142] ACPI: bus type USB registered [ 0.621093] usbcore: registered new interface driver usbfs [ 0.623082] usbcore: registered new interface driver hub [ 0.625091] usbcore: registered new device driver usb [ 0.627206] pps_core: LinuxPPS API ver. 1 registered [ 0.629010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.632061] PTP clock support registered [ 0.634098] EDAC MC: Ver: 3.0.0 [ 0.636215] PCI: Using ACPI for IRQ routing [ 0.638748] NetLabel: Initializing [ 0.640010] NetLabel: domain hash size = 128 [ 0.641010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.643082] NetLabel: unlabeled traffic allowed by default [ 0.646157] vgaarb: loaded [ 0.648265] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.650013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.656337] clocksource: Switched to clocksource kvm-clock [ 0.760194] VFS: Disk quotas dquot_6.6.0 [ 0.762101] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.764782] *** VALIDATE ramfs *** [ 0.766181] *** VALIDATE hugetlbfs *** [ 0.768065] pnp: PnP ACPI init [ 0.770447] pnp: PnP ACPI: found 6 devices [ 0.792049] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.796837] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.799777] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.802406] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.805345] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.808193] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.811291] NET: Registered protocol family 2 [ 0.814371] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.819718] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.823711] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.829643] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.833604] TCP: Hash tables configured (established 65536 bind 65536) [ 0.836678] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.840523] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.843335] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.846344] NET: Registered protocol family 1 [ 0.848779] RPC: Registered named UNIX socket transport module. [ 0.851062] RPC: Registered udp transport module. [ 0.852903] RPC: Registered tcp transport module. [ 0.854831] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.857302] NET: Registered protocol family 44 [ 0.859039] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.861340] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.863547] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.866061] PCI: CLS 0 bytes, default 64 [ 0.868516] Unpacking initramfs... [ 2.257347] debug: unmapping init [mem 0xffff93d7bcc54000-0xffff93d7bffbffff] [ 2.261747] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.264551] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.268060] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.766086] Initialise system trusted keyrings [ 2.767877] Key type blacklist registered [ 2.769461] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.778061] zbud: loaded [ 2.780912] *** VALIDATE nfs *** [ 2.782023] *** VALIDATE nfs4 *** [ 2.783486] pstore: using deflate compression [ 2.786348] Platform Keyring initialized [ 2.883683] NET: Registered protocol family 38 [ 2.886856] Key type asymmetric registered [ 2.889330] Asymmetric key parser 'x509' registered [ 2.891667] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.895402] io scheduler mq-deadline registered [ 2.897425] io scheduler kyber registered [ 2.899502] io scheduler bfq registered [ 2.901871] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.905405] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.908857] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.912291] ACPI: Power Button [PWRF] [ 3.007375] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.099185] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.288618] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.383535] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.571073] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.602911] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.634053] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.638694] Non-volatile memory driver v1.3 [ 3.640274] Linux agpgart interface v0.103 [ 3.669965] virtio_blk virtio1: [vda] 133832 512-byte logical blocks (68.5 MB/65.3 MiB) [ 3.672987] vda: detected capacity change from 0 to 68521984 [ 3.688721] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.691769] vdb: detected capacity change from 0 to 1073741824 [ 3.706511] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.709661] vdc: detected capacity change from 0 to 2621440000 [ 3.725089] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.727925] vdd: detected capacity change from 0 to 2621440000 [ 3.743812] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.747544] vde: detected capacity change from 0 to 4294967296 [ 3.762839] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.765463] vdf: detected capacity change from 0 to 4294967296 [ 3.771855] libphy: Fixed MDIO Bus: probed [ 3.778892] usbcore: registered new interface driver usbserial_generic [ 3.782754] usbserial: USB Serial support registered for generic [ 3.785211] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.789500] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.791458] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.794049] mousedev: PS/2 mouse device common for all mice [ 3.797360] rtc_cmos 00:05: RTC can wake from S4 [ 3.800462] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.800775] rtc_cmos 00:05: registered as rtc0 [ 3.806858] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.810381] intel_pstate: CPU model not supported [ 3.812648] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.817280] hid: raw HID events driver (C) Jiri Kosina [ 3.821024] usbcore: registered new interface driver usbhid [ 3.823182] usbhid: USB HID core driver [ 3.824868] drop_monitor: Initializing network drop monitor service [ 3.827311] Initializing XFRM netlink socket [ 3.829471] NET: Registered protocol family 10 [ 3.833147] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.835818] Segment Routing with IPv6 [ 3.837617] NET: Registered protocol family 17 [ 3.839809] mpls_gso: MPLS GSO support [ 3.846125] RAS: Correctable Errors collector initialized. [ 3.848436] AVX version of gcm_enc/dec engaged. [ 3.849979] AES CTR mode by8 optimization enabled [ 3.914304] sched_clock: Marking stable (3914285449, 0)->(4833026745, -918741296) [ 3.918492] registered taskstats version 1 [ 3.920815] Loading compiled-in X.509 certificates [ 3.923439] zswap: loaded using pool lzo/zbud [ 3.946312] Key type big_key registered [ 3.958934] Key type encrypted registered [ 3.960555] ima: No TPM chip found, activating TPM-bypass! [ 3.962760] ima: Allocated hash algorithm: sha1 [ 3.964644] ima: No architecture policies found [ 3.966431] evm: Initialising EVM extended attributes: [ 3.968320] evm: security.selinux [ 3.969626] evm: security.ima [ 3.970509] evm: security.capability [ 3.971768] evm: HMAC attrs: 0x1 [ 3.974116] rtc_cmos 00:05: setting system clock to 2025-11-16 22:43:15 UTC (1763332995) [ 3.979655] debug: unmapping init [mem 0xffffffff98e03000-0xffffffff98ffffff] [ 3.983124] debug: unmapping init [mem 0xffffffff97b82000-0xffffffff97e58fff] [ 3.991109] Write protecting the kernel read-only data: 28672k [ 3.994306] debug: unmapping init [mem 0xffffffff96203000-0xffffffff963fffff] [ 3.997272] debug: unmapping init [mem 0xffffffff96b14000-0xffffffff96bfffff] [ 4.031174] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.039863] systemd[1]: Detected virtualization kvm. [ 4.041774] systemd[1]: Detected architecture x86-64. [ 4.043772] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.076636] systemd[1]: No hostname configured. [ 4.078424] systemd[1]: Set hostname to . [ 4.080433] random: systemd: uninitialized urandom read (16 bytes read) [ 4.082919] systemd[1]: Initializing machine ID from random generator. [ 4.125778] random: ln: uninitialized urandom read (6 bytes read) [ 4.202131] random: systemd: uninitialized urandom read (16 bytes read) [ 4.205156] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.209568] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.213925] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.776968] device-mapper: uevent: version 1.0.3 [ 4.779098] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.508059] random: fast init done [ 5.510622] virtio_net virtio0 ens2: renamed from eth0 [ 5.593595] scsi host0: ata_piix [ 5.642171] scsi host1: ata_piix [ 5.643618] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.645562] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.291929] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.480711] random: crng init done [ 10.482038] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.949933] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.162859] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.469289] SELinux: Disabled at runtime. [ 12.537596] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.546752] systemd[1]: Detected virtualization kvm. [ 12.548525] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.111725] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.116447] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.123852] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.127887] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.130910] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.139734] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.153204] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 13.212408] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.716294] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.938064] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.984126] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.089526] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.113194] EDAC sbridge: Ver: 1.1.2 [ 16.042496] Key type dns_resolver registered [ 16.343537] NFS: Registering the id_resolver key type [ 16.345285] Key type id_resolver registered [ 16.346960] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg330-server login: [ 65.051083] hrtimer: interrupt took 1937316 ns [ 67.564723] libcfs: loading out-of-tree module taints kernel. [ 67.593620] Key type ._llcrypt registered [ 67.595905] Key type .llcrypt registered [ 67.736487] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 85.603984] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 87.137357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 87.149131] alg: No test for adler32 (adler32-zlib) [ 88.668109] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 89.487684] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 91.279221] Key type lgssc registered [ 92.982867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 106.444842] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 107.324988] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 115.239753] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 122.407793] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 129.328686] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 143.873955] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 155.587602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 155.671897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 155.684187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 156.896747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 156.935739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 157.035469] Lustre: lustre-MDT0000: new disk, initializing [ 157.143540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.163170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 160.998060] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.455388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 173.513526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 173.575524] Lustre: 6519:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 173.606197] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 173.609241] Lustre: Skipped 1 previous similar message [ 173.673866] Lustre: lustre-MDT0001: new disk, initializing [ 173.735575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 173.782325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 173.798084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 177.315314] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.673577] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 190.205515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 190.267472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 190.447596] Lustre: lustre-OST0000: new disk, initializing [ 190.452710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 190.509058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.474896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.480734] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.560792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 195.186872] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.604426] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 208.684507] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 208.787266] Lustre: lustre-OST0001: new disk, initializing [ 208.791566] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 208.849422] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.701780] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.630843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 217.655650] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 217.747351] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 223.816826] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 228.362823] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 238.361347] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing check_logdir /tmp/testlogs/ [ 241.819660] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing yml_node [ 245.189686] Lustre: DEBUG MARKER: Client: 2.16.61.51 [ 247.067756] Lustre: DEBUG MARKER: MDS: 2.16.61.51 [ 249.103732] Lustre: DEBUG MARKER: OSS: 2.16.61.51 [ 250.528458] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 17:47:20 EST 2025 [ 266.008127] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 267.378807] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 281.568360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 281.576857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 281.584490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.127601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.128095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 284.141555] Lustre: Skipped 1 previous similar message [ 284.149639] Lustre: Skipped 1 previous similar message [ 287.539797] LustreError: 12224:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 287.598944] Lustre: server umount lustre-MDT0000 complete [ 290.710781] LustreError: 6511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333282 with bad export cookie 1308438357654514459 [ 290.714045] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 290.731240] LustreError: 6511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 294.369372] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.371300] LustreError: 6524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.372039] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 294.385672] Lustre: Skipped 2 previous similar messages [ 294.413560] LustreError: 6524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 295.393231] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 295.406630] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 300.515792] LustreError: 8435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 300.519272] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 300.566040] Lustre: Skipped 4 previous similar messages [ 305.119719] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 305.240577] LustreError: 12425:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 305.242898] LustreError: 12425:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 305.405902] Lustre: server umount lustre-MDT0001 complete [ 308.886189] LustreError: 12628:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 308.892936] LustreError: 12628:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 308.964634] Lustre: server umount lustre-OST0000 complete [ 312.010879] LustreError: 12828:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 312.013315] LustreError: 12828:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 312.150844] Lustre: server umount lustre-OST0001 complete [ 316.993255] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 323.322311] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 332.323351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 337.935901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 343.709820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 349.469651] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 355.518100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 355.563215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 355.738878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 355.771929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 355.833753] Lustre: lustre-MDT0000: new disk, initializing [ 355.921128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 355.940109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 359.151886] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 368.560206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 368.610942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 368.671989] Lustre: 15015:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 368.733290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 368.736739] Lustre: Skipped 1 previous similar message [ 368.837625] Lustre: lustre-MDT0001: new disk, initializing [ 368.922173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 368.958458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 368.974828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 372.389702] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 378.758577] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 382.198316] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 388.903805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 388.970676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 389.197678] Lustre: lustre-OST0000: new disk, initializing [ 389.205502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 389.290554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 391.074119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 391.082501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 391.264438] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 394.293302] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 400.527211] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 404.409508] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 404.625648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 408.377378] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 408.585669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 411.625463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 411.627945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 411.639099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 411.650196] Lustre: Skipped 1 previous similar message [ 416.643884] LustreError: 17950:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 416.651580] LustreError: 17950:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 416.739486] LustreError: 16742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 416.749562] LustreError: 16742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 416.813349] Lustre: server umount lustre-OST0000 complete [ 420.320479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 420.324574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 420.343307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 420.349345] Lustre: Skipped 1 previous similar message [ 426.438283] LustreError: 18152:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 426.443571] LustreError: 18152:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 426.602065] Lustre: server umount lustre-MDT0000 complete [ 426.978753] LustreError: 17952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 426.998338] LustreError: 17952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 429.760062] LustreError: 15006:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333421 with bad export cookie 1308438357654515992 [ 429.765027] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 429.781677] LustreError: 15006:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 430.058877] Lustre: server umount lustre-MDT0001 complete [ 434.639929] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 17:50:24 (1763333424) [ 436.937848] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 439.043774] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 17:50:28 (1763333428) [ 446.340876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.743153] LustreError: 19023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 446.820846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.611625] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 458.468079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 462.906451] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 469.207389] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 472.638585] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 480.103845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 485.962393] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 491.252744] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 502.240199] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 502.256960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.272641] Lustre: Skipped 2 previous similar messages [ 502.274993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 502.283841] Lustre: Skipped 3 previous similar messages [ 507.655106] LustreError: 21267:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 507.660203] LustreError: 21267:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 507.837600] Lustre: server umount lustre-OST0000 complete [ 526.303128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 526.467907] Lustre: server umount lustre-MDT0000 complete [ 527.841461] LustreError: 19022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.850962] LustreError: 19022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 529.755428] LustreError: 19005:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333521 with bad export cookie 1308438357654516874 [ 529.763654] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.774707] LustreError: 19005:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 530.138417] Lustre: server umount lustre-MDT0001 complete [ 540.236555] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 543.054199] Key type lgssc unregistered [ 543.392914] LNet: 22205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 543.404432] LNetError: 22205:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 543.432490] LNet: Removed LNI 192.168.203.130@tcp [ 544.451193] Key type .llcrypt unregistered [ 544.452921] Key type ._llcrypt unregistered [ 567.672162] Key type ._llcrypt registered [ 567.674346] Key type .llcrypt registered [ 567.760402] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 568.735602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 568.945145] alg: No test for adler32 (adler32-zlib) [ 570.074233] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 570.297648] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 571.943190] Key type lgssc registered [ 572.972628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 580.281069] loop: module loaded [ 580.343742] loop0: detected capacity change from 0 to 204800000 [ 580.548570] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 580.566310] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 580.779353] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 580.830478] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 586.642253] loop0: detected capacity change from 0 to 204800000 [ 586.993735] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 587.011476] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 587.025156] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 587.209275] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 592.696066] loop0: detected capacity change from 0 to 204800000 [ 598.862167] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 598.931954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 598.947459] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 600.265128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 600.318986] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 600.419515] Lustre: lustre-MDT0000: new disk, initializing [ 600.523686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 600.538418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 604.437768] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 612.937774] loop1: detected capacity change from 0 to 204800000 [ 618.367393] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 618.439179] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 618.785968] Lustre: lustre-OST07c6: new disk, initializing [ 618.800909] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 618.803982] Lustre: Skipped 1 previous similar message [ 618.891080] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 624.176320] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 628.322512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 628.340577] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 628.373702] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 634.858742] Lustre: 25870:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 634.869095] Lustre: 25870:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 634.875316] Lustre: 25870:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 634.886649] Lustre: 25870:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 634.894287] Lustre: 25870:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 634.900834] Lustre: 25870:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 655.591931] Lustre: 24368:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 655.602248] Lustre: 24368:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 655.606377] Lustre: 24368:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 655.610098] Lustre: 24368:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 655.613892] Lustre: 24368:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 655.617366] Lustre: 24368:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 659.870565] Lustre: 24369:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 659.902565] Lustre: 24369:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 659.920808] Lustre: 24369:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 659.927224] Lustre: 24369:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.933137] Lustre: 24369:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 659.943230] Lustre: 24369:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.950081] Lustre: 24369:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 659.966983] Lustre: 24369:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.973083] Lustre: 24369:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 659.977561] Lustre: 24369:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.985908] Lustre: 24369:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 659.995702] Lustre: 24369:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.242934] Lustre: 24368:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 663.253648] Lustre: 24368:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 663.258904] Lustre: 24368:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 663.273161] Lustre: 24368:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.283190] Lustre: 24368:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 663.286567] Lustre: 24368:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.306067] Lustre: 24368:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 663.321276] Lustre: 24368:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.335810] Lustre: 24368:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 663.347742] Lustre: 24368:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.355445] Lustre: 24368:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 663.358950] Lustre: 24368:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 667.999098] Lustre: 24369:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 668.020848] Lustre: 24369:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 668.032437] Lustre: 24369:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 668.045931] Lustre: 24369:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.061569] Lustre: 24369:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 668.088843] Lustre: 24369:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.104862] Lustre: 24369:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 668.137032] Lustre: 24369:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.158173] Lustre: 24369:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 668.177764] Lustre: 24369:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.191547] Lustre: 24369:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 668.203334] Lustre: 24369:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.177389] Lustre: 25870:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 679.181791] Lustre: 25870:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 5 previous similar messages [ 679.185806] Lustre: 25870:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 679.206305] Lustre: 25870:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 679.228127] Lustre: 25870:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 679.235887] Lustre: 25870:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 679.239665] Lustre: 25870:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 679.242852] Lustre: 25870:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 679.245750] Lustre: 25870:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 679.249227] Lustre: 25870:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 679.252555] Lustre: 25870:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 679.255223] Lustre: 25870:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 730.592651] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 730.606912] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 730.615773] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 732.289121] LustreError: 26564:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 732.398968] Lustre: server umount lustre-OST07c6 complete [ 741.457966] LustreError: 27065:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 741.461033] LustreError: 27065:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 741.628494] Lustre: server umount lustre-MDT0000 complete [ 756.294617] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 758.929149] Key type lgssc unregistered [ 759.171139] LNet: 27898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.182325] LNetError: 27898:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 759.212219] LNet: Removed LNI 192.168.203.130@tcp [ 760.165214] Key type .llcrypt unregistered [ 760.166900] Key type ._llcrypt unregistered [ 778.681192] Key type ._llcrypt registered [ 778.682717] Key type .llcrypt registered [ 778.794932] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 780.183847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 780.195913] alg: No test for adler32 (adler32-zlib) [ 781.268447] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 781.600107] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 783.423162] Key type lgssc registered [ 784.843044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 791.968690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 791.996551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 793.565888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 797.180707] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 803.810584] LustreError: 29147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 803.818223] LustreError: 29147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 804.762967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 805.254754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 808.951781] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 814.665991] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 817.976439] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 823.499119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 823.835889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.626847] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 833.844658] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 839.302749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 855.015950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.017940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 855.020570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 855.030043] Lustre: Skipped 1 previous similar message [ 855.053624] Lustre: Skipped 1 previous similar message [ 857.210418] LustreError: 31598:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 857.356950] Lustre: server umount lustre-OST0000 complete [ 861.664367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 861.667697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 861.700789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 865.249587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 865.251323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 865.304742] Lustre: Skipped 1 previous similar message [ 867.108833] LustreError: 31800:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 867.113626] LustreError: 31800:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 867.201121] Lustre: server umount lustre-MDT0000 complete [ 870.378743] LustreError: 29143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 870.406080] LustreError: 29143:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 871.217395] LustreError: 29867:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333862 with bad export cookie 8550462906640854112 [ 871.218816] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 871.222539] LustreError: 29867:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 871.417148] LustreError: 32003:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 871.423489] LustreError: 32003:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 871.690081] Lustre: server umount lustre-MDT0001 complete [ 882.720677] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 886.053636] Key type lgssc unregistered [ 886.472779] LNet: 32535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.482301] LNetError: 32535:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.497852] LNet: Removed LNI 192.168.203.130@tcp [ 887.674365] Key type .llcrypt unregistered [ 887.675912] Key type ._llcrypt unregistered [ 907.656569] Key type ._llcrypt registered [ 907.662138] Key type .llcrypt registered [ 907.821593] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 909.360339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 909.558499] alg: No test for adler32 (adler32-zlib) [ 910.657729] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 910.853731] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 912.567327] Key type lgssc registered [ 913.730978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 922.457513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 922.500454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.063065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 927.367561] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 934.369800] LustreError: 33786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.403753] LustreError: 33786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 935.956233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 936.549135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 940.887050] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 946.170470] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 950.157809] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 957.623826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 958.116401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 963.703982] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 969.305198] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 969.401209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 979.940921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 979.942215] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 979.947459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 984.636116] LustreError: 36038:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 984.843491] Lustre: server umount lustre-OST0000 complete [ 990.177298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 990.192795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 990.210104] Lustre: Skipped 2 previous similar messages [ 990.264429] Lustre: Skipped 2 previous similar messages [ 995.145687] LustreError: 36240:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 995.149201] LustreError: 36240:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 995.256552] Lustre: server umount lustre-MDT0000 complete [ 995.297493] LustreError: 33786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 995.304637] LustreError: 33786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 998.657403] LustreError: 33766:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333990 with bad export cookie 3810033858993082429 [ 998.659335] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 998.679466] LustreError: 33766:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 998.832178] LustreError: 36442:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 998.841143] LustreError: 36442:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 999.055399] Lustre: server umount lustre-MDT0001 complete [ 1008.292676] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1010.655835] Key type lgssc unregistered [ 1010.873453] LNet: 36974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.880699] LNetError: 36974:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.891521] LNet: Removed LNI 192.168.203.130@tcp [ 1011.956065] Key type .llcrypt unregistered [ 1011.957904] Key type ._llcrypt unregistered [ 1033.550073] Key type ._llcrypt registered [ 1033.551681] Key type .llcrypt registered [ 1033.657189] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1034.881634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1034.905052] alg: No test for adler32 (adler32-zlib) [ 1035.913849] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 1036.124761] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1037.783572] Key type lgssc registered [ 1038.779939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1045.712811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1045.749221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.280869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1050.692562] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1056.741050] LustreError: 38371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1056.760900] LustreError: 38371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1057.706745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1057.986689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1061.240308] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1065.871278] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1069.021481] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1075.185853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1075.508777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1079.819579] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1084.271894] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1091.883401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1100.581872] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.30@tcp (stopping) [ 1101.596256] LustreError: 40652:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1101.661272] Lustre: server umount lustre-MDT0000 complete [ 1102.307823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1102.308375] LustreError: 38367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.325958] Lustre: Skipped 2 previous similar messages [ 1102.344367] LustreError: 38367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1105.106263] LustreError: 39092:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334096 with bad export cookie 5668595148671440921 [ 1105.107636] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1105.114996] LustreError: 39092:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1105.635846] LustreError: 40491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1105.657983] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.30@tcp (stopping) [ 1105.659548] LustreError: 40491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1107.425787] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.447328] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1110.432751] LustreError: 38366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1110.438431] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1110.458678] LustreError: 38366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1111.270792] LustreError: 40854:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1111.285478] LustreError: 40854:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1111.454662] Lustre: server umount lustre-MDT0001 complete [ 1132.102633] LustreError: 41078:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1132.107224] LustreError: 41078:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1132.219237] Lustre: server umount lustre-OST0000 complete [ 1144.669593] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1147.285496] Key type lgssc unregistered [ 1147.611777] LNet: 41915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1147.636767] LNetError: 41915:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1147.670682] LNet: Removed LNI 192.168.203.130@tcp [ 1148.482181] Key type .llcrypt unregistered [ 1148.483625] Key type ._llcrypt unregistered [ 1169.480552] Key type ._llcrypt registered [ 1169.482316] Key type .llcrypt registered [ 1169.582781] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1170.285876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1170.301826] alg: No test for adler32 (adler32-zlib) [ 1171.276232] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 1171.450733] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1173.104263] Key type lgssc registered [ 1174.143763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1180.915160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1180.945267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1182.400336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1185.612764] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1191.937052] LustreError: 43292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1191.944135] LustreError: 43292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1193.477398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1193.903545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1197.589386] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1204.051703] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1208.091355] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1215.191885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1215.582424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1220.687437] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1225.513846] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1226.875735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1236.219766] Lustre: lustre-OST0000: Not available for connect from 192.168.203.30@tcp (stopping) [ 1237.474369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.477252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1237.478168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1237.487682] Lustre: Skipped 1 previous similar message [ 1240.832489] LustreError: 45538:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1240.940515] Lustre: server umount lustre-OST0000 complete [ 1257.954741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.975484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1257.978074] Lustre: Skipped 1 previous similar message [ 1264.096747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1264.105732] Lustre: Skipped 3 previous similar messages [ 1269.217668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1269.233162] Lustre: Skipped 1 previous similar message [ 1272.287278] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1272.375370] LustreError: 45913:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1272.379181] LustreError: 45913:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1272.448803] Lustre: server umount lustre-MDT0000 complete [ 1274.336343] LustreError: 44004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.342688] LustreError: 44004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1275.673129] LustreError: 44011:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334267 with bad export cookie 9857707129909716176 [ 1275.676387] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1275.685762] LustreError: 44011:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1281.910446] LustreError: 46118:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1281.918459] LustreError: 46118:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1282.037709] Lustre: server umount lustre-MDT0001 complete [ 1289.479247] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1291.586882] Key type lgssc unregistered [ 1291.847485] LNet: 46651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1291.864161] LNetError: 46651:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1292.906559] LNet: Removed LNI 192.168.203.130@tcp [ 1293.582136] Key type .llcrypt unregistered [ 1293.583926] Key type ._llcrypt unregistered [ 1311.950826] Key type ._llcrypt registered [ 1311.952506] Key type .llcrypt registered [ 1312.046817] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1312.892082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1313.029968] alg: No test for adler32 (adler32-zlib) [ 1314.023957] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 1314.199740] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1315.847183] Key type lgssc registered [ 1316.810774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1323.944282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1323.979767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.657575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1329.503572] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1335.784603] LustreError: 48028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.801647] LustreError: 48028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1338.086481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1338.349238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1341.892283] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1346.620995] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1349.289664] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1354.800975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1355.074916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1360.471807] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1365.101150] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1371.504382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1381.215245] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1385.218531] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1397.172570] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-11-1618h06m26s [ 1398.536037] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1401.584705] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.30@tcp (stopping) [ 1402.340790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1402.345699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1402.365776] Lustre: Skipped 2 previous similar messages [ 1402.375451] Lustre: Skipped 2 previous similar messages [ 1403.426618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1403.429296] Lustre: Skipped 3 previous similar messages [ 1406.285068] LustreError: 50615:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1406.369054] Lustre: server umount lustre-MDT0000 complete [ 1407.705062] LustreError: 48022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1409.506254] LustreError: 49073:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1409.540862] LustreError: 49073:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1409.551118] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1413.848990] LustreError: 49073:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1415.797557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.900047] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1416.183147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1416.262651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1417.205594] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1419.652934] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1422.749658] Lustre: DEBUG MARKER: Wait for df (36163) ... [ 1424.319354] Lustre: DEBUG MARKER: done [ 1427.428170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1427.442689] Lustre: Skipped 1 previous similar message [ 1427.444220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1432.546852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1432.553955] Lustre: Skipped 1 previous similar message [ 1441.247137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1441.273827] LustreError: 51672:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1441.277539] LustreError: 51672:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1441.375307] Lustre: server umount lustre-OST0000 complete [ 1447.908756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1447.909451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1447.914310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1447.914320] Lustre: Skipped 3 previous similar messages [ 1447.938624] Lustre: Skipped 1 previous similar message [ 1450.785261] LustreError: 51874:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1450.794653] LustreError: 51874:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1450.890698] Lustre: server umount lustre-MDT0000 complete [ 1454.353270] LustreError: 50345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334445 with bad export cookie 4840073595253541058 [ 1454.360569] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1454.370024] LustreError: 50345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1454.567793] LustreError: 52075:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1454.578288] LustreError: 52075:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1454.826322] Lustre: server umount lustre-MDT0001 complete [ 1463.206366] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1465.952280] Key type lgssc unregistered [ 1466.233855] LNet: 52607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.243579] LNetError: 52607:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.280234] LNet: Removed LNI 192.168.203.130@tcp [ 1466.986548] Key type .llcrypt unregistered [ 1466.988682] Key type ._llcrypt unregistered [ 1497.955622] Key type ._llcrypt registered [ 1497.957942] Key type .llcrypt registered [ 1498.047558] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 1510.303116] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1511.113055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1511.242130] alg: No test for adler32 (adler32-zlib) [ 1512.237208] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 1512.435509] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1514.103147] Key type lgssc registered [ 1514.938346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1521.222503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1526.171637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1530.916468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1535.394878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1540.452343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1540.494598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1540.508565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1541.692814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1541.729869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1541.784782] Lustre: lustre-MDT0000: new disk, initializing [ 1541.873604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1541.891250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1544.802664] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1552.595895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1552.637450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1552.703909] Lustre: 56536:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1552.746855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1552.751137] Lustre: Skipped 1 previous similar message [ 1552.797102] Lustre: lustre-MDT0001: new disk, initializing [ 1552.879304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1552.913597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1552.925197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1555.874022] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1561.954437] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1564.913888] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1570.831230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1570.918814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1571.158294] Lustre: lustre-OST0000: new disk, initializing [ 1571.161426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1571.279696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1576.119528] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1581.131057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1581.140888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1581.236786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1581.897520] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1584.959677] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1585.143239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1588.208669] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1588.405718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1591.779797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1591.785636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1591.790980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1596.156981] LustreError: 59472:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1596.263028] Lustre: server umount lustre-OST0000 complete [ 1598.947621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1598.953367] LustreError: Skipped 1 previous similar message [ 1598.956014] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1598.964624] Lustre: Skipped 1 previous similar message [ 1598.972415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1598.974753] Lustre: Skipped 1 previous similar message [ 1602.019038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1602.019588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1607.136337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1607.140842] Lustre: Skipped 2 previous similar messages [ 1612.257059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1612.263230] Lustre: Skipped 1 previous similar message [ 1613.282213] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1613.349706] LustreError: 59675:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1613.352343] LustreError: 59675:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1613.409731] Lustre: server umount lustre-MDT0000 complete [ 1616.295748] LustreError: 56526:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334607 with bad export cookie 15843386860890037935 [ 1616.302874] LustreError: 56526:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1616.303298] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1616.422512] LustreError: 59879:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1616.429846] LustreError: 59879:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1616.582732] Lustre: server umount lustre-MDT0001 complete [ 1623.462713] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 18:10:13 (1763334613) [ 1628.882202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1629.242149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1632.074875] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1634.273499] LustreError: 60550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1634.299043] LustreError: 60550:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1638.815234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1639.192738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1642.223881] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1646.823582] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1650.021851] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1655.348456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1655.689947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1660.383790] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1664.963403] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1681.125655] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-11-1618h11m10s [ 1682.712420] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1688.939620] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1691.357402] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1691.393097] LustreError: 60546:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1711.892600] Lustre: lustre-MDT0000: Client 116bd732-b4ed-4157-a69b-ff0b6c423296 (at 192.168.203.30@tcp) reconnecting [ 1731.463132] LustreError: 60546:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1733.706457] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1735.844452] Lustre: DEBUG MARKER: done [ 1741.798446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.801933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1741.806596] Lustre: Skipped 1 previous similar message [ 1741.811517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1756.127955] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1756.173445] LustreError: 63583:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1756.191148] LustreError: 63583:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1756.339378] Lustre: server umount lustre-OST0000 complete [ 1762.274890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1762.275674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1762.277060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1762.277065] Lustre: Skipped 5 previous similar messages [ 1762.284755] Lustre: Skipped 1 previous similar message [ 1765.844165] LustreError: 63786:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1765.853860] LustreError: 63786:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1765.940341] Lustre: server umount lustre-MDT0000 complete [ 1769.295829] LustreError: 60532:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334760 with bad export cookie 15843386860890038817 [ 1769.301241] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1769.311840] LustreError: 60532:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1769.653526] Lustre: server umount lustre-MDT0001 complete [ 1778.727028] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1780.845292] Key type lgssc unregistered [ 1781.134905] LNet: 64520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1781.146411] LNetError: 64520:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1781.160920] LNet: Removed LNI 192.168.203.130@tcp [ 1782.030725] Key type .llcrypt unregistered [ 1782.032324] Key type ._llcrypt unregistered [ 1809.868244] Key type ._llcrypt registered [ 1809.870298] Key type .llcrypt registered [ 1809.951622] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 1825.728414] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1826.650656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1826.668567] alg: No test for adler32 (adler32-zlib) [ 1827.773382] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 1828.010333] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1829.682039] Key type lgssc registered [ 1830.676898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1839.562462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1845.666780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1851.639026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1857.137661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1863.767757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1863.867421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1863.893167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.230068] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1865.275436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1865.391768] Lustre: lustre-MDT0000: new disk, initializing [ 1865.541717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1865.577950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1869.034205] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1878.833629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1878.907444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1878.987171] Lustre: 68469:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1879.057829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1879.060935] Lustre: Skipped 1 previous similar message [ 1879.129312] Lustre: lustre-MDT0001: new disk, initializing [ 1879.215986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1879.243591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1879.262231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1882.573028] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1889.663582] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1893.541409] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1899.574796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1899.629431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1899.846222] Lustre: lustre-OST0000: new disk, initializing [ 1899.851091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1899.940885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1904.932661] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1908.306412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1908.313610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1908.390561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1911.970664] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1916.179510] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1916.338071] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1920.525567] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1920.802692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1924.070100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1924.082425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1924.102264] LustreError: Skipped 1 previous similar message [ 1924.112587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1924.133676] Lustre: Skipped 1 previous similar message [ 1928.646155] LustreError: 71407:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1928.860977] Lustre: server umount lustre-OST0000 complete [ 1934.307686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1934.318668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1934.328341] Lustre: Skipped 1 previous similar message [ 1934.330412] Lustre: Skipped 1 previous similar message [ 1939.427026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1939.445032] Lustre: Skipped 2 previous similar messages [ 1939.676906] LustreError: 71608:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1939.689943] LustreError: 71608:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1939.826486] Lustre: server umount lustre-MDT0000 complete [ 1943.575835] LustreError: 68461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334935 with bad export cookie 137831491923737936 [ 1943.593406] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1943.600948] LustreError: 68461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1943.785629] LustreError: 71810:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1943.788937] LustreError: 71810:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1944.028428] Lustre: server umount lustre-MDT0001 complete [ 1951.676510] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 18:15:41 (1763334941) [ 1953.234311] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1954.878190] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 18:15:44 (1763334944) [ 1964.416545] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1967.566737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1967.866720] LustreError: 72859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1967.881015] LustreError: 72859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1967.905514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.570706] Lustre: Failing over lustre-MDT0000 [ 1968.579493] LustreError: 72887:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1968.587853] LustreError: 72887:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1968.592733] LustreError: 72887:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1968.692810] LustreError: 72938:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1968.698531] LustreError: 72938:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1968.794803] Lustre: server umount lustre-MDT0000 complete [ 1973.650305] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 18:16:03 (1763334963) [ 1978.312703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1978.809112] LustreError: 73507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1978.859838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1981.595518] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1983.978540] LustreError: 73507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1987.819936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1988.159309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1990.905424] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1995.223440] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1998.128995] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2003.551663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2009.176448] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2013.464486] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2017.929404] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2023.270243] LustreError: 75854:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2023.276029] LustreError: 75854:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2023.389316] Lustre: server umount lustre-MDT0000 complete [ 2023.411813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.418547] LustreError: 73502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2023.427213] Lustre: Skipped 1 previous similar message [ 2026.494519] LustreError: 73488:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335018 with bad export cookie 137831491923739098 [ 2026.500468] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2026.501415] LustreError: 73488:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2026.830804] Lustre: server umount lustre-MDT0001 complete [ 2029.343744] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2032.686955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2040.349764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2044.895134] Lustre: 66680:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763335020/real 1763335020] req@00000000182b4414 x1848990571236096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763335036 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2044.906817] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.913122] Lustre: Skipped 1 previous similar message [ 2047.967537] Lustre: 66681:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763335023/real 1763335023] req@0000000094501b32 x1848990571236224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763335039 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2049.951215] Lustre: 66682:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763335025/real 1763335025] req@00000000288c167e x1848990571236352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763335041 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2052.066702] LustreError: 66678:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000094501b32 x1848990571237248/t0(0) o250->MGC192.168.203.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2052.510819] LustreError: 75114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2052.517741] LustreError: 75114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2052.593741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2052.599682] Lustre: Skipped 1 previous similar message [ 2052.718556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2053.610839] Lustre: 66679:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763335028/real 1763335028] req@00000000a454f36e x1848990571236608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763335044 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2056.303438] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2057.700649] Lustre: 66680:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763335033/real 1763335033] req@00000000e1b0f33a x1848990571236736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763335049 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2062.860049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2064.232568] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2066.527781] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2071.734554] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2075.640651] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2083.809817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2083.823894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.851443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2088.803902] LustreError: 78312:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2088.811560] LustreError: 78312:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2088.902903] Lustre: server umount lustre-MDT0000 complete [ 2089.960130] LustreError: 76911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2089.979841] LustreError: 76911:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2092.348223] LustreError: 73487:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335083 with bad export cookie 137831491923741002 [ 2092.356954] LustreError: 73487:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2092.364343] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2092.785794] Lustre: server umount lustre-MDT0001 complete [ 2098.661870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2110.214897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.455214] Lustre: 66679:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763335086/real 1763335086] req@00000000c6a715a0 x1848990571262464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763335102 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2111.498877] Lustre: 66679:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2111.512445] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2111.544353] Lustre: Skipped 2 previous similar messages [ 2116.707533] LustreError: 66678:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000026e56b19 x1848990571263488/t0(0) o250->MGC192.168.203.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2117.072586] LustreError: 77392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.132297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.137084] Lustre: Skipped 1 previous similar message [ 2117.174608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2121.054200] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2130.016812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2134.483460] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2135.525072] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2135.549799] Lustre: Skipped 1 previous similar message [ 2139.704883] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2140.154086] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2143.421215] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2149.859141] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.863488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.881943] Lustre: Skipped 1 previous similar message [ 2149.886290] Lustre: Skipped 3 previous similar messages [ 2150.962303] LustreError: 80770:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2150.980295] LustreError: 80770:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2151.023696] Lustre: server umount lustre-MDT0000 complete [ 2154.077596] LustreError: 73488:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335145 with bad export cookie 137831491923742374 [ 2154.086917] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2154.087135] LustreError: 73488:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2154.353943] Lustre: server umount lustre-MDT0001 complete [ 2160.429970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2163.457647] Lustre: DEBUG MARKER: files compared the same [ 2175.692594] Lustre: server umount lustre-OST0000 complete [ 2192.813919] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2195.718742] Key type lgssc unregistered [ 2195.975509] LNet: 82403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2195.981485] LNetError: 82403:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2195.994705] LNet: Removed LNI 192.168.203.130@tcp [ 2196.985858] Key type .llcrypt unregistered [ 2196.988829] Key type ._llcrypt unregistered [ 2219.800155] Key type ._llcrypt registered [ 2219.805311] Key type .llcrypt registered [ 2219.917773] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2220.965608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2220.998979] alg: No test for adler32 (adler32-zlib) [ 2222.130358] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 2222.352730] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2223.991283] Key type lgssc registered [ 2225.377704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2232.795281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2232.861061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2234.398031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2238.190806] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug +malloc all [ 2243.619465] LustreError: 83800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2243.626804] LustreError: 83800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2246.313855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2246.528391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2249.715645] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug +malloc all [ 2254.235467] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2257.444682] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2263.104029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2263.358925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2268.383913] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug +malloc all [ 2273.203867] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2278.762282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2286.413137] LustreError: 86046:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2286.538122] Lustre: server umount lustre-OST0000 complete [ 2292.703936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2292.707723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.728488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2294.240909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.241212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2294.256714] Lustre: Skipped 1 previous similar message [ 2299.361994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2300.394725] LustreError: 86247:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2300.398095] LustreError: 86247:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2300.465113] Lustre: server umount lustre-MDT0000 complete [ 2303.394382] Lustre: 83111:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763335278/real 1763335278] req@000000009b682da4 x1848990984331264/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763335294 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2303.455532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.480863] LustreError: 83796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.500162] LustreError: 83796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2303.820591] LustreError: 83781:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335295 with bad export cookie 5217839647825070280 [ 2303.821288] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2303.841667] LustreError: 83781:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2303.953366] LustreError: 86449:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2303.960727] LustreError: 86449:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2304.142997] Lustre: server umount lustre-MDT0001 complete [ 2312.838681] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2315.216588] Key type lgssc unregistered [ 2315.476342] LNet: 86982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2315.480893] LNetError: 86982:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2315.494922] LNet: Removed LNI 192.168.203.130@tcp [ 2316.238333] Key type .llcrypt unregistered [ 2316.244295] Key type ._llcrypt unregistered [ 2334.325903] Key type ._llcrypt registered [ 2334.327615] Key type .llcrypt registered [ 2334.412748] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2335.055398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2335.111262] alg: No test for adler32 (adler32-zlib) [ 2336.120752] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 2336.362755] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2338.071255] Key type lgssc registered [ 2339.141259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2346.247212] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2346.275126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2361.888776] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2367.007206] LustreError: 88334:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.130@tcp: failed processing log, type 4: rc = -110 [ 2393.759211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.654825] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2403.955430] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2411.079078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2411.229145] Lustre: *** cfs_fail_loc=706, val=0*** [ 2411.514266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2411.621255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2415.851848] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2416.613792] LustreError: 88361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2416.642564] LustreError: 88361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2421.734548] LustreError: 88361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.641224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2425.300463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2429.642334] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2435.526756] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2439.377693] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2442.208123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2442.211092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2442.220445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2444.773036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2444.776806] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2447.285647] LustreError: 90660:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2447.389231] Lustre: server umount lustre-OST0000 complete [ 2450.912554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2450.919772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.936876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.939661] Lustre: Skipped 1 previous similar message [ 2455.011977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2455.013802] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2456.915206] LustreError: 90862:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2456.931956] LustreError: 90862:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2457.031534] Lustre: server umount lustre-MDT0000 complete [ 2460.136950] LustreError: 89312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2460.151429] LustreError: 89312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2460.713482] LustreError: 88341:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335452 with bad export cookie 15499407066929963089 [ 2460.718341] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2460.723109] LustreError: 88341:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2460.892717] LustreError: 91064:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2460.899468] LustreError: 91064:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2461.117709] Lustre: server umount lustre-MDT0001 complete [ 2471.330799] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2473.858804] Key type lgssc unregistered [ 2474.122726] LNet: 91598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2474.128861] LNetError: 91598:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2474.144992] LNet: Removed LNI 192.168.203.130@tcp [ 2474.871624] Key type .llcrypt unregistered [ 2474.874378] Key type ._llcrypt unregistered [ 2494.159625] Key type ._llcrypt registered [ 2494.161276] Key type .llcrypt registered [ 2494.272419] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2495.212501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2495.221940] alg: No test for adler32 (adler32-zlib) [ 2496.250530] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 2496.431507] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2498.063594] Key type lgssc registered [ 2499.001781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2506.554133] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2506.568563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2511.210504] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2518.250097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2518.706655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2522.570612] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2524.128884] LustreError: 93646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2524.150223] LustreError: 93646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2529.256945] LustreError: 93650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2529.288907] LustreError: 93650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2530.999384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2531.496483] LustreError: 93646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.536437] LustreError: 93646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2531.551791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2536.927566] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2536.933306] LustreError: 93649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.962720] LustreError: 93649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2542.052053] LustreError: 93645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.086256] LustreError: 93645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2545.062680] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2545.077941] Lustre: MGS: Client 6af96bdc-3859-4ca8-a61f-6dfa983b045a (at 0@lo) reconnecting [ 2545.400458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2545.418296] LustreError: 93660:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2545.505629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2549.501563] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2564.065328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2564.098370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2569.186455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2569.196577] Lustre: Skipped 2 previous similar messages [ 2569.237669] LustreError: 95610:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2569.391407] Lustre: server umount lustre-OST0000 complete [ 2574.305261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.313133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2574.319959] Lustre: Skipped 2 previous similar messages [ 2574.329725] Lustre: Skipped 1 previous similar message [ 2578.898203] LustreError: 95811:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2578.905120] LustreError: 95811:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2578.941620] Lustre: server umount lustre-MDT0000 complete [ 2579.430510] LustreError: 93650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.451292] LustreError: 93650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2582.397202] LustreError: 96013:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2582.404367] LustreError: 96013:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2582.604950] Lustre: server umount lustre-MDT0001 complete [ 2585.885858] LustreError: 96215:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2585.889487] LustreError: 96215:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2585.948630] Lustre: server umount MGS complete [ 2595.064672] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:26:25 (1763335585) [ 2619.058324] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 2625.634114] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2634.422481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2640.947129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2647.482111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2653.499505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2659.467076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2659.522718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2663.007146] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2671.913841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2672.008921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2672.177727] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2672.248977] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2672.305613] Lustre: lustre-MDT0001: new disk, initializing [ 2672.388280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2672.433046] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2672.438921] Lustre: 100735:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2672.444179] LustreError: 100735:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2675.568955] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2684.295894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2684.346633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2684.507497] Lustre: lustre-OST0000: new disk, initializing [ 2684.510507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2684.560275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2688.188555] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2692.141243] Lustre: 101989:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000d24afbbe x1848991271657088/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763335699 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2693.884600] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2699.602257] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2699.623986] Lustre: MGS: Client 70cf5b37-90a1-4da0-814b-4e32add8e90f (at 0@lo) reconnecting [ 2699.669444] Lustre: 100020:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2699.698417] LustreError: 100020:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2699.736161] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2699.812524] Lustre: lustre-MDT0000: new disk, initializing [ 2699.953306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2699.969814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2701.026960] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2701.038914] Lustre: 102447:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000d9b97975 x1848991271662848/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1763335708 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2703.295557] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2710.126725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2710.143511] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2710.233059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2733.027384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2733.032474] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2733.033128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2733.039141] Lustre: Skipped 1 previous similar message [ 2737.274555] LustreError: 102969:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2737.280689] LustreError: 102969:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2737.420483] Lustre: server umount lustre-OST0000 complete [ 2743.266793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2743.267471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2743.280823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2743.289398] Lustre: Skipped 1 previous similar message [ 2743.296839] Lustre: Skipped 2 previous similar messages [ 2746.534600] LustreError: 103170:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2746.536794] LustreError: 103170:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2746.568109] Lustre: server umount lustre-MDT0000 complete [ 2748.386354] LustreError: 103173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2748.402843] LustreError: 103173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2755.676322] Lustre: server umount lustre-MDT0001 complete [ 2758.608664] Lustre: server umount MGS complete [ 2766.838778] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:29:17 (1763335757) [ 2777.715735] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2779.500663] Key type lgssc unregistered [ 2779.711828] LNet: 104910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2779.718386] LNetError: 104910:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2779.728794] LNet: Removed LNI 192.168.203.130@tcp [ 2780.232150] Key type .llcrypt unregistered [ 2780.234166] Key type ._llcrypt unregistered [ 2792.909582] Key type ._llcrypt registered [ 2792.911392] Key type .llcrypt registered [ 2792.977830] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2793.688249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2793.803857] alg: No test for adler32 (adler32-zlib) [ 2794.794204] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 2794.992508] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2796.671434] Key type lgssc registered [ 2797.533415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2803.797120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2803.818629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2805.242662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2808.133819] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2814.302600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2814.631529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2818.591451] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2823.203663] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2826.300296] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2831.482844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2831.704450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2835.834452] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2840.098502] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2851.176546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2860.514808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2860.522779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.535362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2861.538935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2861.540632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2861.540751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2861.543187] Lustre: Skipped 1 previous similar message [ 2865.652604] LustreError: 108357:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2865.741969] Lustre: server umount lustre-OST0000 complete [ 2871.268797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2871.274900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2871.282366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2874.757440] LustreError: 108559:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2874.760673] LustreError: 108559:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2874.844598] Lustre: server umount lustre-MDT0000 complete [ 2876.898515] LustreError: 106091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2876.909323] LustreError: 106091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2877.193782] LustreError: 106072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335868 with bad export cookie 17551773728191826486 [ 2877.195461] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2877.203209] LustreError: 106072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2877.294428] LustreError: 108760:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2877.298234] LustreError: 108760:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2877.417995] Lustre: server umount lustre-MDT0001 complete [ 2884.222948] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2886.439583] Key type lgssc unregistered [ 2886.690796] LNet: 109291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2886.703155] LNetError: 109291:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2886.712649] LNet: Removed LNI 192.168.203.130@tcp [ 2887.353752] Key type .llcrypt unregistered [ 2887.355693] Key type ._llcrypt unregistered [ 2898.515523] Key type ._llcrypt registered [ 2898.517029] Key type .llcrypt registered [ 2898.611974] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2899.105478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2899.138066] alg: No test for adler32 (adler32-zlib) [ 2900.097336] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 2900.254788] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2901.887407] Key type lgssc registered [ 2902.703263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2907.411856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2907.418657] LustreError: 110334:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2907.435525] LustreError: 110335:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2907.441315] LustreError: 110334:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4981 [ 2907.451285] LustreError: 110334:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2907.461224] LustreError: 110334:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2907.470859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2908.915088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2914.564173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2914.853277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2918.330771] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2922.164944] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing lsmod [ 2924.459651] LustreError: 111507:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2924.489454] LustreError: 111514:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2924.493981] LustreError: 111507:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4969 [ 2924.502779] LustreError: 111507:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2924.511170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2924.515697] LustreError: 111507:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2924.806424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2930.148917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2930.156350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2930.179573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2931.172869] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2931.173663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2931.186152] Lustre: Skipped 1 previous similar message [ 2931.196261] Lustre: Skipped 2 previous similar messages [ 2933.025786] LustreError: 111703:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2933.107568] Lustre: server umount lustre-MDT0000 complete [ 2936.079363] LustreError: 110355:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335927 with bad export cookie 4058186150510081016 [ 2936.086724] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2936.096809] LustreError: 110355:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2936.234957] LustreError: 111906:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2936.239563] LustreError: 111906:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2936.296151] LustreError: 110370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2936.296156] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2936.320475] LustreError: 110370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2936.428766] Lustre: server umount lustre-MDT0001 complete [ 2945.088994] LustreError: 112109:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2945.102064] LustreError: 112109:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2945.230909] Lustre: server umount lustre-OST0000 complete [ 2953.783823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.337095] LustreError: 112585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2954.429807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2958.208785] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2959.858968] LustreError: 112585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2963.938291] LustreError: 112585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.372921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2966.771874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2970.272507] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2974.551804] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2977.117568] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2981.691190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2981.974270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2986.496518] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2990.365995] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2999.401640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3004.898764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3004.899045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3004.918643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3004.924242] Lustre: Skipped 1 previous similar message [ 3006.209619] LustreError: 114853:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3006.216278] LustreError: 114853:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3006.304237] Lustre: server umount lustre-OST0000 complete [ 3010.016413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.021437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.023288] Lustre: Skipped 2 previous similar messages [ 3010.032764] Lustre: Skipped 1 previous similar message [ 3015.137347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3015.145166] Lustre: Skipped 1 previous similar message [ 3015.322612] LustreError: 115053:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3015.325718] LustreError: 115053:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3015.396905] Lustre: server umount lustre-MDT0000 complete [ 3018.869379] LustreError: 112568:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336010 with bad export cookie 4058186150510081716 [ 3018.873070] LustreError: 112568:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3018.882328] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3019.194973] Lustre: server umount lustre-MDT0001 complete [ 3027.655979] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 3030.157794] Key type lgssc unregistered [ 3030.558646] LNet: 115787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3030.563160] LNetError: 115787:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3030.579608] LNet: Removed LNI 192.168.203.130@tcp [ 3031.385231] Key type .llcrypt unregistered [ 3031.388903] Key type ._llcrypt unregistered [ 3049.024824] Key type ._llcrypt registered [ 3049.026488] Key type .llcrypt registered [ 3049.106469] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3049.599138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3049.662570] alg: No test for adler32 (adler32-zlib) [ 3050.662210] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 3050.856400] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 3052.495300] Key type lgssc registered [ 3053.282811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3058.547257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3058.563890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.894537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3062.430996] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3067.795748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3068.013970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3070.124471] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3073.251187] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3075.445419] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3079.811235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3080.024422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3083.944688] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3085.189185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3087.971149] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3096.771957] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3099.034356] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3103.714193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3103.716782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.724296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3103.724880] Lustre: Skipped 1 previous similar message [ 3107.704561] LustreError: 119509:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3107.794540] Lustre: server umount lustre-OST0000 complete [ 3113.952728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3113.954278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3113.960137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3113.967041] Lustre: Skipped 1 previous similar message [ 3113.969934] Lustre: Skipped 1 previous similar message [ 3116.314152] LustreError: 119715:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3116.317821] LustreError: 119715:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3116.384829] Lustre: server umount lustre-MDT0000 complete [ 3118.998055] LustreError: 117146:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336110 with bad export cookie 1408494490925876709 [ 3119.003851] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3119.128484] LustreError: 119917:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3119.131491] LustreError: 119917:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3119.254959] Lustre: server umount lustre-MDT0001 complete [ 3125.884539] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 3127.750719] Key type lgssc unregistered [ 3127.949769] LNet: 120447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3127.956781] LNetError: 120447:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3127.965679] LNet: Removed LNI 192.168.203.130@tcp [ 3128.446924] Key type .llcrypt unregistered [ 3128.448519] Key type ._llcrypt unregistered [ 3138.522928] Key type ._llcrypt registered [ 3138.525081] Key type .llcrypt registered [ 3138.574809] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3139.061803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3139.101069] alg: No test for adler32 (adler32-zlib) [ 3140.029616] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 3140.158682] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 3141.775179] Key type lgssc registered [ 3142.480079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3147.309728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3147.323229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3148.636561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3151.100567] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3157.028236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3157.262020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3159.832752] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3164.051744] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3166.602293] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3170.832379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3171.066099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3174.131573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3175.103955] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3178.720740] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3189.731761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3189.737526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3189.749041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3192.802878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3192.805141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3192.806757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3197.925196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3197.934924] Lustre: Skipped 2 previous similar messages [ 3202.015172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3202.059283] LustreError: 123871:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3202.139477] Lustre: server umount lustre-OST0000 complete [ 3208.168387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3208.173397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3208.174606] Lustre: Skipped 1 previous similar message [ 3208.195243] Lustre: Skipped 1 previous similar message [ 3210.934988] LustreError: 124074:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3210.939625] LustreError: 124074:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3211.039175] Lustre: server umount lustre-MDT0000 complete [ 3213.280201] LustreError: 121622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.309421] LustreError: 121622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3214.037197] LustreError: 123243:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336205 with bad export cookie 8289171341964059984 [ 3214.038406] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3214.049050] LustreError: 123243:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3214.134807] LustreError: 124275:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3214.138134] LustreError: 124275:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3214.267298] Lustre: server umount lustre-MDT0001 complete [ 3222.064892] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 3224.212422] Key type lgssc unregistered [ 3224.444660] LNet: 124806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3224.456114] LNetError: 124806:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3224.476496] LNet: Removed LNI 192.168.203.130@tcp [ 3225.163518] Key type .llcrypt unregistered [ 3225.170105] Key type ._llcrypt unregistered [ 3245.792145] Key type ._llcrypt registered [ 3245.793828] Key type .llcrypt registered [ 3245.910754] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3247.237450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3247.331095] alg: No test for adler32 (adler32-zlib) [ 3248.358853] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 3248.584743] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 3250.287174] Key type lgssc registered [ 3251.460660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3258.063364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3258.080152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3259.420403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3262.336357] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3268.141752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3268.445977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3271.504424] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3275.384569] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3278.100212] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3283.668883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3283.809724] LustreError: 127795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.821726] LustreError: 127795:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3283.926557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3287.799821] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3289.067480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3291.581254] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3298.117899] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3300.879889] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3300.886905] Lustre: Skipped 2 previous similar messages [ 3304.519971] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3304.578286] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3304.698461] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3304.713214] Lustre: Skipped 2 previous similar messages [ 3310.134479] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3310.137027] Lustre: Skipped 1 previous similar message [ 3317.119708] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.30@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3317.126114] Lustre: Skipped 2 previous similar messages [ 3325.997835] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.30@tcp 192.168.0.[2,10]@tcp [ 3329.919291] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.30/24@tcp in log lustre-MDT0000 [ 3329.930490] Lustre: Skipped 2 previous similar messages [ 3331.174494] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.30/24@tcp [ 3331.179251] Lustre: Skipped 1 previous similar message [ 3338.727777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3338.728807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3338.736722] LustreError: Skipped 1 previous similar message [ 3338.752299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3339.238974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3339.241628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3339.243564] LustreError: Skipped 2 previous similar messages [ 3339.256696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3339.259019] Lustre: Skipped 1 previous similar message [ 3342.401139] LustreError: 129899:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3342.524745] Lustre: server umount lustre-OST0000 complete [ 3349.474249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.475542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3349.483852] Lustre: Skipped 1 previous similar message [ 3349.492117] Lustre: Skipped 1 previous similar message [ 3351.624518] LustreError: 130101:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3351.627411] LustreError: 130101:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3351.716175] Lustre: server umount lustre-MDT0000 complete [ 3354.595311] LustreError: 126190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.605574] LustreError: 126190:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3355.248843] LustreError: 126177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336346 with bad export cookie 2230276479198497703 [ 3355.249611] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3355.260621] LustreError: 126177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3355.436072] LustreError: 130303:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3355.457052] LustreError: 130303:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3355.682957] Lustre: server umount lustre-MDT0001 complete [ 3365.874216] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 3368.499356] Key type lgssc unregistered [ 3368.772291] LNet: 130835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3368.781508] LNetError: 130835:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3368.799189] LNet: Removed LNI 192.168.203.130@tcp [ 3369.651873] Key type .llcrypt unregistered [ 3369.655115] Key type ._llcrypt unregistered [ 3391.035033] Key type ._llcrypt registered [ 3391.036598] Key type .llcrypt registered [ 3391.105841] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3391.930991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3391.940979] alg: No test for adler32 (adler32-zlib) [ 3392.940217] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 3393.194833] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 3394.911263] Key type lgssc registered [ 3395.894667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3402.275216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3402.298942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3403.784660] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3403.806687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3403.825256] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.30/24@tcp [ 3407.135197] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3413.985019] LustreError: 132355:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3413.992410] LustreError: 132355:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3414.024326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3414.377631] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3414.404385] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.30/24@tcp [ 3414.452505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3418.234551] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3422.635715] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3425.488273] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3429.814321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3430.043144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3434.128333] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3437.418484] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3448.503536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3459.044834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3459.046571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.052437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3459.056192] Lustre: Skipped 1 previous similar message [ 3459.062377] Lustre: Skipped 1 previous similar message [ 3464.164642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3464.192433] LustreError: 134680:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3464.275700] Lustre: server umount lustre-OST0000 complete [ 3469.286434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3469.287990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3469.297105] Lustre: Skipped 1 previous similar message [ 3469.299985] Lustre: Skipped 1 previous similar message [ 3473.425524] LustreError: 134881:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3473.429090] LustreError: 134881:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3473.488670] Lustre: server umount lustre-MDT0000 complete [ 3474.401182] LustreError: 132355:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3474.425248] LustreError: 132355:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3475.933066] LustreError: 132335:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336467 with bad export cookie 17907118340083574017 [ 3475.938839] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3475.945719] LustreError: 132335:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3476.047195] LustreError: 135082:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3476.051382] LustreError: 135082:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3476.157472] Lustre: server umount lustre-MDT0001 complete [ 3484.081599] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 3486.335389] Key type lgssc unregistered [ 3486.595526] LNet: 135614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3486.610297] LNetError: 135614:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3486.635855] LNet: Removed LNI 192.168.203.130@tcp [ 3487.326537] Key type .llcrypt unregistered [ 3487.331824] Key type ._llcrypt unregistered [ 3508.412118] Key type ._llcrypt registered [ 3508.413960] Key type .llcrypt registered [ 3508.459659] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 3517.599745] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3518.097503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3518.161131] alg: No test for adler32 (adler32-zlib) [ 3519.145361] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 3519.303814] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 3520.985500] Key type lgssc registered [ 3522.148913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3530.032849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3535.920408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3540.228194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3544.517249] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3548.685522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3548.725873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3548.739857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3549.889878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3549.912817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3549.963952] Lustre: lustre-MDT0000: new disk, initializing [ 3550.025430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3550.036334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3552.180654] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3559.164319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3559.219689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3559.277995] Lustre: 138968:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3559.319247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3559.322259] Lustre: Skipped 1 previous similar message [ 3559.376885] Lustre: lustre-MDT0001: new disk, initializing [ 3559.457956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3559.482136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3559.486970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3562.165224] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3568.211204] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3570.602564] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3575.299546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3575.337798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3575.553859] Lustre: lustre-OST0000: new disk, initializing [ 3575.556696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3575.621715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.013446] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3583.543204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3583.549663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3583.598226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3585.402672] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3588.578332] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3588.738158] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3591.723992] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3591.899665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3593.708632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3593.709893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3593.712863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3593.722144] Lustre: Skipped 1 previous similar message [ 3598.816255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3598.819315] Lustre: Skipped 2 previous similar messages [ 3599.425332] LustreError: 141906:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3599.539747] Lustre: server umount lustre-OST0000 complete [ 3603.942053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3603.971446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3603.983107] Lustre: Skipped 1 previous similar message [ 3608.844787] LustreError: 142107:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3608.849677] LustreError: 142107:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3608.919214] Lustre: server umount lustre-MDT0000 complete [ 3612.945020] LustreError: 138960:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336604 with bad export cookie 8764309886933850960 [ 3612.953270] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3613.124582] LustreError: 142309:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3613.130331] LustreError: 142309:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3613.288966] Lustre: server umount lustre-MDT0001 complete [ 3622.358923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3622.933492] LustreError: 142786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3622.958632] LustreError: 142786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3623.005232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3626.699696] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3628.016253] LustreError: 142786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3633.124388] LustreError: 142786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3633.184676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3633.461620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3636.523505] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3641.700489] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3644.647838] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3650.147370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3650.596383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3655.617908] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3659.815300] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3662.957297] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3668.506707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 3671.988830] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3672.203240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3679.867616] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3679.946374] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3680.083285] Lustre: lustre-OST0001: new disk, initializing [ 3680.087464] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3680.199755] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3684.680303] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3690.071150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3690.078963] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3690.166224] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3690.931611] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3691.132444] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3694.877472] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3695.080139] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3697.847076] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e2b0318d800.ost_server_uuid 50 [ 3699.526550] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e2b0318d800.ost_server_uuid in FULL state after 0 sec [ 3705.291165] LustreError: 147067:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3705.294311] LustreError: 147067:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3705.353706] Lustre: server umount lustre-OST0001 complete [ 3705.827923] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3705.832415] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3705.843085] LustreError: Skipped 1 previous similar message [ 3705.843801] LustreError: 144382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3705.870564] Lustre: Skipped 2 previous similar messages [ 3708.896494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3708.899235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3708.909468] LustreError: Skipped 1 previous similar message [ 3708.910764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3708.927727] Lustre: Skipped 1 previous similar message [ 3710.952030] LustreError: 144383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3710.966628] LustreError: 144383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3714.555661] LustreError: 147269:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3714.560256] LustreError: 147269:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3714.646502] Lustre: server umount lustre-OST0000 complete [ 3720.674822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3720.686099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3720.702481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3720.705192] Lustre: Skipped 3 previous similar messages [ 3723.239837] LustreError: 147470:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3723.248180] LustreError: 147470:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3723.349469] Lustre: server umount lustre-MDT0000 complete [ 3726.035422] LustreError: 142766:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336717 with bad export cookie 8764309886933851849 [ 3726.043597] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3726.053139] LustreError: 142766:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3726.336244] Lustre: server umount lustre-MDT0001 complete [ 3739.329525] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 3741.505141] Key type lgssc unregistered [ 3741.793931] LNet: 148662:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3741.802373] LNetError: 148662:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3741.816548] LNet: Removed LNI 192.168.203.130@tcp [ 3742.508914] Key type .llcrypt unregistered [ 3742.511301] Key type ._llcrypt unregistered [ 3770.306770] Key type ._llcrypt registered [ 3770.308218] Key type .llcrypt registered [ 3770.378429] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 3782.446985] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3783.307207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3783.319100] alg: No test for adler32 (adler32-zlib) [ 3784.334994] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 3784.530402] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 3786.175627] Key type lgssc registered [ 3787.264837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3795.855984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3802.026222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3807.033097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3812.795788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3819.327817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3819.384203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3819.396418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.569077] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3820.604729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3820.700931] Lustre: lustre-MDT0000: new disk, initializing [ 3820.776795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3820.798634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3824.219328] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3833.476931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3833.532038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3833.636858] Lustre: 152590:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3833.702307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3833.706313] Lustre: Skipped 1 previous similar message [ 3833.777163] Lustre: lustre-MDT0001: new disk, initializing [ 3833.898853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3833.936799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3833.949192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3837.305554] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3844.421113] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3847.560960] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3852.760514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3852.817982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3852.989928] Lustre: lustre-OST0000: new disk, initializing [ 3852.995489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3853.061164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3857.963425] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3860.019104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3860.024750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3860.123705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3864.745262] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3868.304773] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3868.489704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3871.851197] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3872.040123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3875.812162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3875.820428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3875.824673] LustreError: Skipped 1 previous similar message [ 3875.835265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3875.846844] Lustre: Skipped 1 previous similar message [ 3879.737974] LustreError: 155532:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3879.987483] Lustre: server umount lustre-OST0000 complete [ 3885.026790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3885.030506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3885.051311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3885.058933] Lustre: Skipped 1 previous similar message [ 3886.052704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3886.055594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3891.172147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3891.174944] Lustre: Skipped 1 previous similar message [ 3896.289199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3896.292479] Lustre: Skipped 1 previous similar message [ 3898.335126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3898.438580] LustreError: 155733:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3898.447706] LustreError: 155733:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3898.548353] Lustre: server umount lustre-MDT0000 complete [ 3902.129507] LustreError: 152582:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336893 with bad export cookie 13923932643987616116 [ 3902.132077] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3902.154981] LustreError: 152582:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3902.302399] LustreError: 155936:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3902.306761] LustreError: 155936:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3902.498242] Lustre: server umount lustre-MDT0001 complete [ 3910.649737] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 18:48:20 (1763336900) [ 3925.787931] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 3933.585937] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3940.801850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3947.247643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3954.186823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3959.442370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3965.694179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3965.751400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3965.955964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3965.997370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3966.050914] Lustre: lustre-MDT0000: new disk, initializing [ 3966.118221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3966.135674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3969.527807] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3978.773729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3978.849750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3978.949840] Lustre: 158927:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3978.994908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3978.998258] Lustre: Skipped 1 previous similar message [ 3979.050185] Lustre: lustre-MDT0001: new disk, initializing [ 3979.129881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3979.156783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3979.171660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3982.925821] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3989.961866] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3993.260703] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3999.529315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3999.599380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3999.817375] Lustre: lustre-OST0000: new disk, initializing [ 3999.828701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3999.942633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4004.810393] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4006.952226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4006.963039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4007.142668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4011.420871] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4015.699084] Lustre: Failing over lustre-OST0000 [ 4015.759792] LustreError: 161369:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4015.766609] LustreError: 161369:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4015.876108] Lustre: server umount lustre-OST0000 complete [ 4034.016267] Lustre: 150806:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763337009/real 1763337009] req@00000000d7323fd3 x1848992623302272/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1763337025 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4034.021414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4034.060765] Lustre: 150806:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4034.278320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4034.589098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4034.599865] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4039.137471] Lustre: 150805:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763337014/real 1763337014] req@0000000006d8b206 x1848992623303424/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763337030 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4039.166805] Lustre: 150805:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4040.070528] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4043.744334] Lustre: 150806:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763337019/real 1763337019] req@000000001cc5f358 x1848992623304448/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763337035 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4043.755482] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4043.816710] Lustre: 150806:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4043.940192] Lustre: Failing over lustre-MDT0000 [ 4043.946073] LustreError: 150802:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4044.029773] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.30@tcp (stopping) [ 4044.044864] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4044.046403] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4044.178950] LustreError: 162350:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4044.182209] LustreError: 162350:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4044.325466] Lustre: server umount lustre-MDT0000 complete [ 4045.799285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4045.802676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4045.836854] Lustre: Skipped 1 previous similar message [ 4045.849048] LustreError: 158940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4045.887938] LustreError: 158940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4048.868741] LustreError: 158940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4048.887338] LustreError: 158940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4049.167942] Lustre: 150803:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763337024/real 1763337024] req@00000000f34d2005 x1848992623305600/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763337040 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4053.984562] LustreError: 158935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4054.020589] LustreError: 158935:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4059.108259] LustreError: 158940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4059.131512] LustreError: 158940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4064.230295] LustreError: 158940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4064.259756] LustreError: 158940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4064.506186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4064.775637] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4064.935728] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4065.077450] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4065.548023] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4068.775378] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4070.390276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4070.416200] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4070.482816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4076.001035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4076.008461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4076.023677] Lustre: Skipped 2 previous similar messages [ 4076.030126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4080.097871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4081.725573] LustreError: 163237:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4081.738516] LustreError: 163237:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4081.895945] Lustre: server umount lustre-OST0000 complete [ 4091.550390] Lustre: server umount lustre-MDT0000 complete [ 4094.821688] LustreError: 160673:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337086 with bad export cookie 13923932643987618223 [ 4094.826501] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4094.834313] LustreError: 160673:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4095.116356] Lustre: server umount lustre-MDT0001 complete [ 4104.672919] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 4107.770110] Key type lgssc unregistered [ 4108.193847] LNet: 164176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4108.211305] LNetError: 164176:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4108.234676] LNet: Removed LNI 192.168.203.130@tcp [ 4109.369591] Key type .llcrypt unregistered [ 4109.381848] Key type ._llcrypt unregistered [ 4143.916824] Key type ._llcrypt registered [ 4143.918399] Key type .llcrypt registered [ 4144.021442] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 4145.234822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4145.262699] alg: No test for adler32 (adler32-zlib) [ 4146.237429] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 4146.468572] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 4148.167266] Key type lgssc registered [ 4148.998605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4155.387826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4155.410522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4156.884387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4160.274340] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4166.514350] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4167.137852] LustreError: 165680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4167.151446] LustreError: 165680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4170.126558] LustreError: 165709:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4170.135642] LustreError: 165709:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4170.141678] LustreError: 165709:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 0, failed: rc = -5 [ 4170.249167] LustreError: 166233:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4170.434577] Lustre: server umount lustre-MDT0000 complete [ 4183.913888] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 4192.455966] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 4201.619773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4207.801503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4213.625707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4219.018122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4224.903692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4224.939972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.147834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4225.187099] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4225.267123] Lustre: lustre-MDT0000: new disk, initializing [ 4225.351253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4225.379882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4228.581739] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4238.645066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4238.725386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4238.837362] Lustre: 168877:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4238.897232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4238.899788] Lustre: Skipped 1 previous similar message [ 4239.006190] Lustre: lustre-MDT0001: new disk, initializing [ 4239.141731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4239.227966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4239.245874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4243.297967] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4250.315265] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4253.305936] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4258.942210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4258.980414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4259.169711] Lustre: lustre-OST0000: new disk, initializing [ 4259.175197] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4259.272406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4264.319678] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4268.623709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4268.645351] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4268.761775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4271.808320] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4274.922462] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4276.398553] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:54:26 (1763337266) [ 4489.184207] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4489.186881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4489.200083] Lustre: Skipped 1 previous similar message [ 4489.210936] Lustre: Skipped 2 previous similar messages [ 4492.909570] LustreError: 172399:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4492.918206] LustreError: 172399:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4492.990773] Lustre: server umount lustre-MDT0000 complete [ 4494.303930] LustreError: 168889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4494.311562] LustreError: 168889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4496.122719] LustreError: 168869:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337487 with bad export cookie 13932489186185057385 [ 4496.127387] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4496.137361] LustreError: 168869:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4499.424470] LustreError: 168884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.425816] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4499.440775] Lustre: Skipped 1 previous similar message [ 4499.447473] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4501.471825] LustreError: 168883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4501.475773] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4502.305914] LustreError: 172601:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4502.312591] LustreError: 172601:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4502.500603] Lustre: server umount lustre-MDT0001 complete [ 4512.245055] LustreError: 172803:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4512.248421] LustreError: 172803:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4512.371659] Lustre: server umount lustre-OST0000 complete [ 4519.783756] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 4527.308430] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 4535.552828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4542.450440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4548.877270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4554.725762] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4561.151527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4561.211751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4561.445193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4561.484923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4561.545479] Lustre: lustre-MDT0000: new disk, initializing [ 4561.647654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4561.669376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4565.220917] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4574.008445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4574.059972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4574.126590] Lustre: 175145:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4574.174069] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4574.177321] Lustre: Skipped 1 previous similar message [ 4574.261239] Lustre: lustre-MDT0001: new disk, initializing [ 4574.365315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4574.449662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4574.464096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4577.549883] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4583.631303] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4586.263720] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4591.805683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4591.872382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4592.032454] Lustre: lustre-OST0000: new disk, initializing [ 4592.035280] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4592.129504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4596.770956] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4601.390902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4601.401697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4601.477880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4603.206314] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4606.516758] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4606.674418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4609.829857] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4609.996631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4611.557723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4611.560274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4611.565706] Lustre: Skipped 1 previous similar message [ 4616.678890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4616.684654] Lustre: Skipped 2 previous similar messages [ 4617.459187] LustreError: 178083:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4617.473641] LustreError: 178083:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4617.596972] Lustre: server umount lustre-OST0000 complete [ 4620.768139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4620.771672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4626.549798] LustreError: 178284:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4626.552586] LustreError: 178284:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4626.675535] Lustre: server umount lustre-MDT0000 complete [ 4626.932508] LustreError: 175155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4626.940784] LustreError: 175155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4629.581913] LustreError: 176900:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337621 with bad export cookie 13932489186185216215 [ 4629.584093] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4629.589989] LustreError: 176900:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4629.834244] Lustre: server umount lustre-MDT0001 complete [ 4636.232586] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:00:26 (1763337626) [ 4648.493776] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 4656.865398] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 4665.701895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4671.738578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4677.149749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4683.018175] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4689.083257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4689.134418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4689.378175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4689.420273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4689.494584] Lustre: lustre-MDT0000: new disk, initializing [ 4689.588524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4689.605068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4692.921771] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4701.569505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4701.624887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4701.708945] Lustre: 181488:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4702.021283] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4705.471170] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4712.676865] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4716.204797] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4722.907047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4722.959297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4723.123386] Lustre: lustre-OST0000: new disk, initializing [ 4723.125170] Lustre: Skipped 1 previous similar message [ 4723.134754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4723.137699] Lustre: Skipped 2 previous similar messages [ 4728.658666] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4730.939241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4730.951321] Lustre: Skipped 1 previous similar message [ 4730.956161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4731.019303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4735.347231] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4741.608637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4741.608977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4741.609604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4741.609609] Lustre: Skipped 3 previous similar messages [ 4741.640322] Lustre: Skipped 2 previous similar messages [ 4742.781168] LustreError: 184033:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4742.785712] LustreError: 184033:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4742.882851] Lustre: server umount lustre-OST0000 complete [ 4752.169415] Lustre: server umount lustre-MDT0000 complete [ 4754.863023] LustreError: 181479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337746 with bad export cookie 13932489186185217090 [ 4754.865692] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4754.873208] LustreError: 181479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4755.109413] Lustre: server umount lustre-MDT0001 complete [ 4762.674918] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:02:32 (1763337752) [ 4776.198903] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 4784.865286] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 4793.809875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4799.900662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4804.883957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4810.387569] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4816.119957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4816.198627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.445767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4816.477407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4816.530263] Lustre: lustre-MDT0000: new disk, initializing [ 4816.693951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4816.697203] Lustre: Skipped 2 previous similar messages [ 4816.707906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4820.146041] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4830.002897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4830.103551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4830.207311] Lustre: 187426:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4830.410203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4833.931845] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4840.866168] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4844.412711] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4851.485859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4851.573407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4857.549950] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4862.131570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4864.647729] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4872.674557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4872.676711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4872.696922] Lustre: Skipped 2 previous similar messages [ 4872.700027] LustreError: Skipped 1 previous similar message [ 4872.706684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4872.714603] Lustre: Skipped 5 previous similar messages [ 4876.691183] LustreError: 189976:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4876.704485] LustreError: 189976:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4876.863693] Lustre: server umount lustre-OST0000 complete [ 4881.888443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4886.611273] Lustre: server umount lustre-MDT0000 complete [ 4888.039619] LustreError: 187435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4888.058104] LustreError: 187435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4889.964273] LustreError: 187419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337881 with bad export cookie 13932489186185218175 [ 4889.969850] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4889.980081] LustreError: 187419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4890.320455] Lustre: server umount lustre-MDT0001 complete [ 4898.996622] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 4901.478770] Key type lgssc unregistered [ 4901.766957] LNet: 190912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4901.772264] LNetError: 190912:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4902.821651] LNet: Removed LNI 192.168.203.130@tcp [ 4903.651337] Key type .llcrypt unregistered [ 4903.652792] Key type ._llcrypt unregistered [ 4925.785678] Key type ._llcrypt registered [ 4925.787236] Key type .llcrypt registered [ 4925.900149] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 4926.942482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4926.968576] alg: No test for adler32 (adler32-zlib) [ 4928.116741] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 4928.345247] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 4929.983154] Key type lgssc registered [ 4931.382770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4939.621221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4939.637911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4941.119790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4945.183871] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4950.497903] LustreError: 192312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4950.532957] LustreError: 192312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4953.504313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4953.915903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4957.903351] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4963.145724] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4966.577384] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4973.014491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4973.330346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4979.089696] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4984.392371] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4985.647443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5006.306190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5006.314864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5006.326620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5006.335074] Lustre: Skipped 1 previous similar message [ 5011.429482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5011.433985] Lustre: Skipped 2 previous similar messages [ 5011.578160] LustreError: 194581:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5011.720991] Lustre: server umount lustre-OST0000 complete [ 5015.523942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5015.535370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5015.548931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5016.554993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.565472] LustreError: 194782:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5021.573554] LustreError: 194782:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5021.666864] LustreError: 192306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5021.700646] LustreError: 192306:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5021.709688] Lustre: server umount lustre-MDT0000 complete [ 5026.065637] LustreError: 192292:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338017 with bad export cookie 5952265326512046641 [ 5026.070339] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5026.209493] LustreError: 192292:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5026.331078] LustreError: 194984:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5026.336543] LustreError: 194984:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5026.539883] Lustre: server umount lustre-MDT0001 complete [ 5035.318890] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 5037.502182] Key type lgssc unregistered [ 5037.778587] LNet: 195516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5037.783414] LNetError: 195516:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5037.799750] LNet: Removed LNI 192.168.203.130@tcp [ 5038.455423] Key type .llcrypt unregistered [ 5038.457996] Key type ._llcrypt unregistered [ 5060.211939] Key type ._llcrypt registered [ 5060.213582] Key type .llcrypt registered [ 5060.311749] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5061.318880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5061.329491] alg: No test for adler32 (adler32-zlib) [ 5062.360944] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 5062.566554] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 5064.215185] Key type lgssc registered [ 5065.312801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5072.970683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5072.986794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.443372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5078.812292] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5083.619337] LustreError: 196917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5083.638631] LustreError: 196917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5087.351482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5087.699731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5091.220816] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5095.601171] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5098.673805] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5105.128024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5105.379791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5111.273759] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5116.986178] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5118.703979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5128.162577] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5128.171266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5128.183981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5128.943788] Lustre: lustre-OST0000: Not available for connect from 192.168.203.30@tcp (stopping) [ 5129.193423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5129.197253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5133.896711] LustreError: 199164:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5134.067971] Lustre: server umount lustre-OST0000 complete [ 5139.815969] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8e2b0759e800.ost_server_uuid 50 [ 5141.486895] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5158.883812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5158.886091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5158.899025] Lustre: Skipped 1 previous similar message [ 5158.926880] Lustre: Skipped 3 previous similar messages [ 5160.954852] LustreError: 199533:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5160.961439] LustreError: 199533:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5161.039308] Lustre: server umount lustre-MDT0000 complete [ 5164.004030] LustreError: 196917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5164.025366] LustreError: 196917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5164.964992] LustreError: 196898:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338156 with bad export cookie 16546673042513974881 [ 5164.965830] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5164.981695] LustreError: 196898:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5165.154344] LustreError: 199736:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5165.160253] LustreError: 199736:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5165.383959] Lustre: server umount lustre-MDT0001 complete [ 5173.641522] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 19:09:23 (1763338163) [ 5179.299851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5179.599751] LustreError: 200408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5179.669073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5183.693659] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5184.994026] LustreError: 200408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5190.116568] LustreError: 200408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5192.294622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5192.668111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5196.638554] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5202.422434] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5206.103689] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5213.230194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5213.655052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5219.018836] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5224.137924] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5225.031825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5231.469212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5231.539219] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5231.747939] Lustre: lustre-OST0001: new disk, initializing [ 5231.775874] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5231.918351] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5237.867181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5237.880426] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5238.024346] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5238.381781] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5246.104173] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5250.533039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5250.537076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5250.552684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5253.600174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5254.721118] LustreError: 203615:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5254.741764] LustreError: 203615:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5254.828453] Lustre: server umount lustre-OST0000 complete [ 5256.924284] LustreError: 202009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5261.315874] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5261.531690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5265.194846] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5265.378407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5274.086083] LustreError: 202009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5274.115337] LustreError: 202009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5279.204603] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5279.207549] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5279.219122] LustreError: Skipped 1 previous similar message [ 5279.222369] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5279.235525] Lustre: Skipped 2 previous similar messages [ 5279.244728] Lustre: Skipped 4 previous similar messages [ 5285.131286] LustreError: 204347:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5285.140802] LustreError: 204347:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5285.319744] Lustre: server umount lustre-OST0001 complete [ 5289.447602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5289.451519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5289.467885] Lustre: Skipped 1 previous similar message [ 5289.470386] Lustre: Skipped 2 previous similar messages [ 5295.238464] LustreError: 204548:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5295.243810] LustreError: 204548:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5295.353818] Lustre: server umount lustre-MDT0000 complete [ 5299.059331] LustreError: 200389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338290 with bad export cookie 16546673042513975952 [ 5299.067677] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5299.082432] LustreError: 200389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5299.496062] Lustre: server umount lustre-MDT0001 complete [ 5305.811126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5310.139956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5314.068546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5317.178685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5322.263467] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 19:11:52 (1763338312) [ 5328.833984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5329.129915] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5329.157957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5329.324836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5333.280939] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5342.188303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5342.303269] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5342.332296] Lustre: 206243:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5346.741519] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5352.684760] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5356.726224] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5363.428874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5363.764224] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5369.855510] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5369.972436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5375.200791] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5381.428122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5381.614601] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5387.509974] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5388.922199] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5392.686332] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5395.426032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5395.429236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5395.443582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5395.446453] Lustre: Skipped 3 previous similar messages [ 5409.247151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5409.285424] LustreError: 209440:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5409.298595] LustreError: 209440:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5409.457826] Lustre: server umount lustre-OST0000 complete [ 5409.765031] LustreError: 207870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5409.782744] LustreError: 207870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5424.610905] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5424.619903] LustreError: Skipped 1 previous similar message [ 5430.250562] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5430.253384] Lustre: Skipped 12 previous similar messages [ 5430.686281] Lustre: server umount lustre-OST0001 complete [ 5435.360445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5435.373312] Lustre: Skipped 4 previous similar messages [ 5441.333907] Lustre: server umount lustre-MDT0000 complete [ 5444.960210] LustreError: 206235:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338436 with bad export cookie 16546673042513977359 [ 5444.961984] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5444.973398] LustreError: 206235:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5445.124756] LustreError: 210068:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5445.127985] LustreError: 210068:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5445.322895] Lustre: server umount lustre-MDT0001 complete [ 5451.038790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5455.111833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5459.177498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5463.234733] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5470.345733] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 19:14:20 (1763338460) [ 5486.646495] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 5494.784321] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5505.481969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5511.961539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5517.816935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5525.324566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5532.259009] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5532.333926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5532.527945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5532.569299] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5532.719329] Lustre: lustre-MDT0000: new disk, initializing [ 5532.835636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5532.838880] Lustre: Skipped 3 previous similar messages [ 5532.851325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5537.380306] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5548.602760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5548.684862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5548.763160] Lustre: 213895:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5548.853134] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5548.856322] Lustre: Skipped 1 previous similar message [ 5549.016839] Lustre: lustre-MDT0001: new disk, initializing [ 5549.219413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5549.244930] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5554.273903] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5561.485845] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5564.943214] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5571.011934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5571.104451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5571.310901] Lustre: lustre-OST0000: new disk, initializing [ 5571.319401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5576.641909] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5579.855562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5579.861670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5579.979481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5583.025319] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5586.834382] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5587.101165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5590.985350] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5591.167837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5595.617265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5595.617847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5595.629931] LustreError: Skipped 2 previous similar messages [ 5595.639666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5595.648391] Lustre: Skipped 5 previous similar messages [ 5607.391625] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5607.436592] LustreError: 216836:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5607.444586] LustreError: 216836:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5607.572986] Lustre: server umount lustre-OST0000 complete [ 5626.339185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5626.626784] Lustre: server umount lustre-MDT0000 complete [ 5630.495482] LustreError: 213887:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338622 with bad export cookie 16546673042513979151 [ 5630.498362] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5630.526792] LustreError: 213887:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5631.022593] Lustre: server umount lustre-MDT0001 complete [ 5640.264205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5641.003351] LustreError: 217720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5641.024672] LustreError: 217720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5644.918581] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5654.292745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5658.355778] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5663.697228] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5666.805502] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5673.391576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5673.687620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5673.697246] Lustre: Skipped 4 previous similar messages [ 5679.581435] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5685.451377] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5689.410119] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5689.661741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5693.573205] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5693.828568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5697.521337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5697.531793] LustreError: Skipped 1 previous similar message [ 5710.303347] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5710.507454] Lustre: server umount lustre-OST0000 complete [ 5717.213995] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5721.883155] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5751.472585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5757.911623] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5763.186706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.192192] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.197874] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5763.801978] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5768.539405] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 5773.793683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5773.800871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5773.806334] Lustre: Skipped 6 previous similar messages [ 5773.813793] Lustre: Skipped 17 previous similar messages [ 5777.986034] LustreError: 222106:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5777.988768] LustreError: 222106:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 5778.112168] Lustre: server umount lustre-OST0000 complete [ 5787.973618] Lustre: server umount lustre-MDT0000 complete [ 5789.164094] LustreError: 217715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5789.173857] LustreError: 217715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5791.603904] LustreError: 217700:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338783 with bad export cookie 16546673042513980033 [ 5791.606322] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5791.608607] LustreError: 217700:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5791.907475] Lustre: server umount lustre-MDT0001 complete [ 5800.485111] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 19:19:50 (1763338790) [ 5807.420377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5812.045110] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5820.872747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5825.626700] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5831.753594] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5835.452606] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5842.148354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5847.474229] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5852.715866] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5857.190263] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5857.413985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5861.115226] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5861.348030] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5867.605257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5867.710083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5867.886461] Lustre: lustre-OST0001: new disk, initializing [ 5867.890717] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5873.678896] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5874.288034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5874.297032] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5874.408651] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5880.953965] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5884.995899] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5885.202830] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5889.000271] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5889.181884] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5895.136098] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5895.139396] LustreError: Skipped 2 previous similar messages [ 5905.375489] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5905.457812] Lustre: server umount lustre-OST0001 complete [ 5910.204479] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5910.378551] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5913.855884] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5914.064518] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5942.761796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5942.981896] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5942.984166] Lustre: Skipped 5 previous similar messages [ 5948.409422] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5948.458939] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5948.463951] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5948.466653] Lustre: Skipped 1 previous similar message [ 5948.550965] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5953.394567] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5954.881395] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5962.616450] Lustre: server umount lustre-OST0001 complete [ 5983.629819] Lustre: server umount lustre-OST0000 complete [ 5993.294224] Lustre: server umount lustre-MDT0000 complete [ 5996.346393] LustreError: 223162:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338987 with bad export cookie 16546673042513980992 [ 5996.351972] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5996.356798] LustreError: 223162:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6002.729970] Lustre: server umount lustre-MDT0001 complete [ 6007.544292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6010.846902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6014.171703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6017.866095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6024.388232] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 19:23:34 (1763339014) [ 6030.182842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6030.375538] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6030.407427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6034.415641] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6041.160868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6041.218444] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6041.241960] Lustre: 231170:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6044.502302] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6049.503703] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6052.622865] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6057.884397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6058.021280] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6063.068870] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6068.359758] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6077.419172] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6077.489143] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6082.615455] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6087.716981] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6089.247322] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6093.040334] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6093.266535] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6097.356830] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6097.668345] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6101.750860] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e2b1819f800.ost_server_uuid 50 [ 6103.279683] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e2b1819f800.ost_server_uuid in FULL state after 0 sec [ 6104.186650] Lustre: Permanently deactivating lustre-OST0001 [ 6104.199527] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6106.515329] Lustre: Permanently reactivating lustre-OST0001 [ 6113.767526] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6113.768776] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6113.782937] Lustre: Skipped 11 previous similar messages [ 6113.786741] Lustre: Skipped 16 previous similar messages [ 6115.256545] LustreError: 235155:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6115.265024] LustreError: 235155:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 6115.330607] Lustre: server umount lustre-OST0001 complete [ 6118.883220] LustreError: 232794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6118.899699] LustreError: 232794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 34 previous similar messages [ 6133.215188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6133.372727] Lustre: server umount lustre-OST0000 complete [ 6142.599665] Lustre: server umount lustre-MDT0000 complete [ 6145.507228] LustreError: 231162:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339137 with bad export cookie 16546673042513982287 [ 6145.512750] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6145.789394] Lustre: server umount lustre-MDT0001 complete [ 6150.441077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6153.865266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6157.827704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6161.505769] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6167.270829] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 19:25:57 (1763339157) [ 6168.158857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6174.158465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6174.727146] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6174.766721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6174.771849] Lustre: Skipped 5 previous similar messages [ 6179.248207] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6187.564497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6187.675769] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6187.719670] Lustre: 237313:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6191.063795] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6196.580190] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6200.038892] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6206.298217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6206.366807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6206.514548] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6206.570347] Lustre: Permanently deactivating lustre-OST0000 [ 6206.587258] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6212.062384] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6212.662057] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6217.175206] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6222.593567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6222.700791] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6228.685794] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6231.779748] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6234.096933] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6235.641387] Lustre: Permanently reactivating lustre-OST0000 [ 6235.661797] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6235.668035] Lustre: Skipped 2 previous similar messages [ 6240.825053] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6240.830317] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.846427] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6240.855941] Lustre: Skipped 1 previous similar message [ 6240.873163] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6240.884893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6245.864406] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 6245.868366] LustreError: Skipped 4 previous similar messages [ 6251.627848] Lustre: server umount lustre-OST0001 complete [ 6260.985870] Lustre: server umount lustre-OST0000 complete [ 6270.255593] Lustre: server umount lustre-MDT0000 complete [ 6273.891400] LustreError: 240468:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339265 with bad export cookie 16546673042513983939 [ 6273.895275] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6273.900778] LustreError: 240468:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6274.258818] Lustre: server umount lustre-MDT0001 complete [ 6284.295978] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 6287.525657] Key type lgssc unregistered [ 6287.906059] LNet: 241716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6287.921131] LNetError: 241716:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6288.936728] LNet: Removed LNI 192.168.203.130@tcp [ 6289.533388] Key type .llcrypt unregistered [ 6289.534598] Key type ._llcrypt unregistered [ 6308.582705] Key type ._llcrypt registered [ 6308.589617] Key type .llcrypt registered [ 6308.712906] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 6309.474486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6309.493896] alg: No test for adler32 (adler32-zlib) [ 6310.439850] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 6310.606110] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 6312.255371] Key type lgssc registered [ 6313.102122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6315.673019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6320.938237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6320.962374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6322.377493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6325.426596] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6331.874552] LustreError: 243146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6331.892185] LustreError: 243146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6332.510480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6332.606694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6332.744328] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6332.767418] Lustre: 243136:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6332.775809] Lustre: Permanently deactivating lustre-MDT0001 [ 6332.787946] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6333.074981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6336.279596] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6337.183859] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6337.186678] LustreError: 243177:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6337.198923] LustreError: 243177:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6337.204387] LustreError: 243177:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 6342.165836] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6345.395188] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6351.482148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6351.722989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6357.314478] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6357.864265] LustreError: 244764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6357.885382] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 6361.978922] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6363.106897] LustreError: 245134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6367.209347] LustreError: 244765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6367.222383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6367.237225] LustreError: 244765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6367.401925] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6367.621310] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6372.439577] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6372.849092] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6372.850076] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6376.518655] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6379.893465] Lustre: Permanently reactivating lustre-MDT0001 [ 6379.905191] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6379.909791] Lustre: Skipped 2 previous similar messages [ 6386.982499] Lustre: lustre-MDT0001: Client af1aa050-ff6a-479a-990f-c8edb995272a (at 192.168.203.30@tcp) reconnecting [ 6390.574849] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6392.142891] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6396.053273] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6396.213640] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6399.930347] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6400.087816] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6402.170532] Lustre: Permanently deactivating lustre-MDT0001 [ 6402.191630] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6402.225176] Lustre: Skipped 2 previous similar messages [ 6410.837505] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6414.817082] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6414.819581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6414.840191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6414.850645] Lustre: Skipped 3 previous similar messages [ 6419.937913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6419.947511] Lustre: Skipped 3 previous similar messages [ 6420.333490] LustreError: 247216:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6420.405527] Lustre: server umount lustre-MDT0000 complete [ 6423.107830] LustreError: 243126:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339414 with bad export cookie 10078896377899258849 [ 6423.113565] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6423.121655] LustreError: 243126:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6423.237380] LustreError: 247418:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6423.241072] LustreError: 247418:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6423.335413] Lustre: server umount lustre-MDT0001 complete [ 6432.144144] LustreError: 247618:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6432.152698] LustreError: 247618:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6432.184074] Lustre: server umount lustre-OST0000 complete [ 6441.515546] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 19:30:31 (1763339431) [ 6449.115482] LustreError: 248623:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6449.117899] LustreError: 248623:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6449.188178] Lustre: server umount lustre-OST0001 complete [ 6453.103781] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 6459.880803] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 6467.527728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6473.586440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6477.986983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6482.621706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6487.797689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6487.842375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6488.003177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6488.011522] Lustre: Skipped 2 previous similar messages [ 6488.041339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6488.096746] Lustre: lustre-MDT0000: new disk, initializing [ 6488.174463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6488.202792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6491.445777] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6500.609574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6500.654656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6500.734953] Lustre: 250809:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6500.784277] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6500.787078] Lustre: Skipped 1 previous similar message [ 6500.840307] Lustre: lustre-MDT0001: new disk, initializing [ 6500.922266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6500.941413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6500.954794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6504.106562] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6510.542511] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6514.168151] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6520.550034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6520.589720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6520.767939] Lustre: lustre-OST0000: new disk, initializing [ 6520.775664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6520.845077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6525.843303] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6529.580854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6529.597118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6529.714465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6532.569831] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6534.280964] LustreError: 250817:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6540.752199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6540.807514] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6540.934682] Lustre: lustre-OST0001: new disk, initializing [ 6540.938207] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6544.367123] LustreError: 250817:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6544.379968] LustreError: 250817:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6544.995253] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6547.544633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6547.560609] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6547.645492] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6551.072707] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6552.227363] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6554.471132] LustreError: 250817:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6556.925896] Lustre: lustre-OST0001: Not available for connect from 192.168.203.30@tcp (stopping) [ 6558.182456] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6558.185186] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6558.197571] Lustre: Skipped 4 previous similar messages [ 6561.912128] LustreError: 254301:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6561.919188] LustreError: 254301:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6561.998841] Lustre: server umount lustre-OST0001 complete [ 6562.036311] LustreError: 252540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6573.541494] LustreError: 253514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6573.550767] LustreError: 253514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6578.664558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6578.666651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6578.671818] Lustre: Skipped 1 previous similar message [ 6578.676549] Lustre: Skipped 3 previous similar messages [ 6583.227112] LustreError: 254525:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6583.235736] LustreError: 254525:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6583.318680] Lustre: server umount lustre-OST0000 complete [ 6587.874082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6587.877232] LustreError: Skipped 1 previous similar message [ 6587.879401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6587.886225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6592.938570] Lustre: server umount lustre-MDT0000 complete [ 6594.018320] LustreError: 250820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6594.035668] LustreError: 250820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6595.596263] LustreError: 250802:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339587 with bad export cookie 10078896377899261887 [ 6595.597605] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6595.607234] LustreError: 250802:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6601.858971] LustreError: 254929:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6601.870192] LustreError: 254929:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6602.142874] Lustre: server umount lustre-MDT0001 complete [ 6611.945390] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 6614.450378] Key type lgssc unregistered [ 6614.699176] LNet: 255461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6614.706346] LNetError: 255461:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6615.718162] LNet: Removed LNI 192.168.203.130@tcp [ 6616.374505] Key type .llcrypt unregistered [ 6616.376267] Key type ._llcrypt unregistered [ 6644.155648] Key type ._llcrypt registered [ 6644.159698] Key type .llcrypt registered [ 6644.266499] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 6657.147337] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 6657.753144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6657.769517] alg: No test for adler32 (adler32-zlib) [ 6658.764181] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 6659.014427] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 6660.703198] Key type lgssc registered [ 6661.719798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6668.331696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6673.645892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6678.371421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6682.786322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6687.551804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6687.596296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6687.608538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6688.786056] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6688.836686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6688.902993] Lustre: lustre-MDT0000: new disk, initializing [ 6688.974419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6689.011299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6692.085990] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6700.100339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6700.155210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6700.213731] Lustre: 259414:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6700.281854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6700.285494] Lustre: Skipped 1 previous similar message [ 6700.348259] Lustre: lustre-MDT0001: new disk, initializing [ 6700.416229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6700.454033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6700.470381] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6703.483409] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6708.760407] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6711.649923] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6716.644791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6716.701876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6716.886512] Lustre: lustre-OST0000: new disk, initializing [ 6716.890068] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6716.940720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6721.588608] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6726.704808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6726.721616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6726.785656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6727.325996] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6730.428166] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6730.582700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6733.617375] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6733.784606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6736.879862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6736.892809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6736.894546] Lustre: Skipped 1 previous similar message [ 6736.901162] Lustre: Skipped 1 previous similar message [ 6741.570141] LustreError: 262353:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6741.679223] Lustre: server umount lustre-OST0000 complete [ 6746.592114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6746.600078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6746.612640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6750.890105] LustreError: 262555:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6750.894262] LustreError: 262555:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6750.965152] Lustre: server umount lustre-MDT0000 complete [ 6753.793593] LustreError: 259406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339745 with bad export cookie 16334765917626020791 [ 6753.794281] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6753.799765] LustreError: 259406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6753.918360] LustreError: 262756:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6753.925207] LustreError: 262756:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6754.071333] Lustre: server umount lustre-MDT0001 complete [ 6760.172141] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 19:35:50 (1763339750) [ 6764.858768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6765.144294] LustreError: 263427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6765.153686] LustreError: 263427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6765.203457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6767.655114] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6770.656846] LustreError: 263427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6774.343520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6774.582727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6777.449753] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6782.355720] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6786.499460] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6792.975143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6793.351689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6798.209227] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6802.900244] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6862.306400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6862.311289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6862.320965] LustreError: Skipped 1 previous similar message [ 6862.326473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6862.335510] Lustre: Skipped 2 previous similar messages [ 6862.344289] Lustre: Skipped 3 previous similar messages [ 6865.848413] LustreError: 265774:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6865.862599] LustreError: 265774:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6865.963958] Lustre: server umount lustre-OST0000 complete [ 6869.362783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6878.969672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6879.292370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6882.352439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6882.360427] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6882.365656] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6882.371696] Lustre: Skipped 1 previous similar message [ 6884.099621] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6888.585241] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6928.355403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6928.356588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6928.378831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6933.477505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6933.484416] Lustre: Skipped 2 previous similar messages [ 6933.899135] LustreError: 267440:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6933.915341] LustreError: 267440:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6934.041874] Lustre: server umount lustre-OST0000 complete [ 6938.594416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6938.602926] Lustre: Skipped 2 previous similar messages [ 6938.610402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6943.618372] LustreError: 267644:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6943.631174] LustreError: 267644:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6943.701658] Lustre: server umount lustre-MDT0000 complete [ 6943.726390] LustreError: 263427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6946.980723] LustreError: 263407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339938 with bad export cookie 16334765917626021666 [ 6946.982359] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6946.993027] LustreError: 263407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6947.253510] Lustre: server umount lustre-MDT0001 complete [ 6954.804883] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 6956.998622] Key type lgssc unregistered [ 6957.218920] LNet: 268377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6957.224866] LNetError: 268377:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6957.241819] LNet: Removed LNI 192.168.203.130@tcp [ 6957.795077] Key type .llcrypt unregistered [ 6957.796654] Key type ._llcrypt unregistered [ 6976.957755] Key type ._llcrypt registered [ 6976.959853] Key type .llcrypt registered [ 6977.059774] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 6977.885307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6977.899055] alg: No test for adler32 (adler32-zlib) [ 6978.879879] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 6979.058662] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 6980.695437] Key type lgssc registered [ 6981.632761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6988.909818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6988.935437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6990.250113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6993.590856] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6999.523834] LustreError: 269754:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6999.556401] LustreError: 269754:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7000.915271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7001.190662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7004.407135] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7009.238190] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7012.428587] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7017.791579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7018.022610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7022.468474] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7026.710444] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7035.445736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7055.845524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7055.846941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7055.856555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7055.885862] Lustre: Skipped 1 previous similar message [ 7060.960942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7060.973373] Lustre: Skipped 1 previous similar message [ 7061.129358] LustreError: 272794:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7061.215352] Lustre: server umount lustre-OST0000 complete [ 7066.082511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7066.108210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7066.111785] Lustre: Skipped 1 previous similar message [ 7071.201130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7071.209671] Lustre: Skipped 1 previous similar message [ 7071.388502] LustreError: 272995:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7071.391826] LustreError: 272995:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7071.469661] Lustre: server umount lustre-MDT0000 complete [ 7074.964827] LustreError: 269736:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340066 with bad export cookie 13166622889367116165 [ 7074.976568] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7074.981722] LustreError: 269736:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7075.161895] LustreError: 273197:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7075.165126] LustreError: 273197:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7075.427661] Lustre: server umount lustre-MDT0001 complete [ 7085.299711] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 7088.014185] Key type lgssc unregistered [ 7088.344961] LNet: 273729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7088.359362] LNetError: 273729:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7089.381830] LNet: Removed LNI 192.168.203.130@tcp [ 7090.308675] Key type .llcrypt unregistered [ 7090.310703] Key type ._llcrypt unregistered [ 7107.258418] Key type ._llcrypt registered [ 7107.259869] Key type .llcrypt registered [ 7107.381414] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7108.055714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7108.098307] alg: No test for adler32 (adler32-zlib) [ 7109.314627] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 7109.522339] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 7111.167180] Key type lgssc registered [ 7112.402423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7120.216438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7120.260176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7121.881320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7125.383831] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7132.129576] LustreError: 274931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7132.155883] LustreError: 274931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7134.494596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7134.986982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7138.959959] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7145.100736] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7149.486861] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7156.484814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7156.742835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7162.621225] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7167.562606] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7167.995700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7178.210299] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7178.214390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7178.218333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7178.265660] Lustre: Skipped 1 previous similar message [ 7183.330100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7183.351499] Lustre: Skipped 2 previous similar messages [ 7183.453161] LustreError: 277322:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7183.605807] Lustre: server umount lustre-OST0000 complete [ 7188.450722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7188.454105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7188.481829] Lustre: Skipped 1 previous similar message [ 7188.502177] Lustre: Skipped 1 previous similar message [ 7193.568756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7198.689219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7198.691947] Lustre: Skipped 2 previous similar messages [ 7202.271179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7202.373745] LustreError: 277523:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7202.377290] LustreError: 277523:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7202.440698] Lustre: server umount lustre-MDT0000 complete [ 7203.810293] LustreError: 274926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7203.825931] LustreError: 274926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7206.275969] LustreError: 274911:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340197 with bad export cookie 7054570394835192677 [ 7206.305405] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7206.451034] LustreError: 277726:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7206.454398] LustreError: 277726:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7206.701714] Lustre: server umount lustre-MDT0001 complete [ 7217.331309] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 7219.923520] Key type lgssc unregistered [ 7220.200431] LNet: 278260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7220.213196] LNetError: 278260:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7220.247477] LNet: Removed LNI 192.168.203.130@tcp [ 7221.022206] Key type .llcrypt unregistered [ 7221.027609] Key type ._llcrypt unregistered [ 7242.234923] Key type ._llcrypt registered [ 7242.236571] Key type .llcrypt registered [ 7242.390834] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7243.538781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7243.613631] alg: No test for adler32 (adler32-zlib) [ 7244.799313] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 7245.063679] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 7246.775197] Key type lgssc registered [ 7248.165235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7257.902683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7257.918836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7259.547506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7264.138093] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7269.855993] LustreError: 279657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7269.884389] LustreError: 279657:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7270.892184] LustreError: 279657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7271.899362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7272.245059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7276.014385] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7281.511276] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7285.675589] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7292.310315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7292.635877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7297.747840] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7302.738921] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7305.998537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7331.814425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7331.829940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7331.834083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7331.853180] Lustre: Skipped 1 previous similar message [ 7334.208603] LustreError: 282679:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7334.347957] Lustre: server umount lustre-OST0000 complete [ 7338.978373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7338.996067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7339.019073] Lustre: Skipped 1 previous similar message [ 7339.031575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7342.048925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7342.049672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7343.938934] LustreError: 282881:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7343.942703] LustreError: 282881:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7344.103705] Lustre: server umount lustre-MDT0000 complete [ 7347.171467] LustreError: 279657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7347.182358] LustreError: 279657:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7347.585599] LustreError: 279639:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340339 with bad export cookie 16243491248359055732 [ 7347.588165] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7347.596498] LustreError: 279639:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7347.725546] LustreError: 283084:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7347.729335] LustreError: 283084:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7347.879516] Lustre: server umount lustre-MDT0001 complete [ 7357.620740] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 7360.349139] Key type lgssc unregistered [ 7360.651773] LNet: 283615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7360.659766] LNetError: 283615:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7360.675729] LNet: Removed LNI 192.168.203.130@tcp [ 7361.297433] Key type .llcrypt unregistered [ 7361.303204] Key type ._llcrypt unregistered [ 7377.402219] Key type ._llcrypt registered [ 7377.404172] Key type .llcrypt registered [ 7377.508654] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7378.358193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7378.448280] alg: No test for adler32 (adler32-zlib) [ 7379.508280] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 7379.700801] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 7381.397431] Key type lgssc registered [ 7382.719290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7388.872123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7388.900438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.285480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7393.335958] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7400.326670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7400.416382] LustreError: 284844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7400.436431] LustreError: 284844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7400.600353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7403.495795] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7408.266872] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7411.445432] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7417.451555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7417.792365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7423.476659] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7428.808450] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7435.199978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7450.595739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7450.601427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7450.606934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7451.518162] LustreError: 287267:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7451.644847] Lustre: server umount lustre-OST0000 complete [ 7455.714239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7455.724192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7455.733796] Lustre: Skipped 1 previous similar message [ 7455.737386] Lustre: Skipped 2 previous similar messages [ 7460.837629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7460.846229] Lustre: Skipped 2 previous similar messages [ 7461.626769] LustreError: 287469:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7461.636768] LustreError: 287469:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7461.782633] Lustre: server umount lustre-MDT0000 complete [ 7465.303873] LustreError: 284796:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340456 with bad export cookie 14822726328223377967 [ 7465.306984] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7465.323925] LustreError: 284796:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7465.469451] LustreError: 287670:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7465.478366] LustreError: 287670:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7465.638534] Lustre: server umount lustre-MDT0001 complete [ 7475.964745] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 7478.971505] Key type lgssc unregistered [ 7479.303844] LNet: 288203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7479.321731] LNetError: 288203:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7479.336559] LNet: Removed LNI 192.168.203.130@tcp [ 7480.377967] Key type .llcrypt unregistered [ 7480.380341] Key type ._llcrypt unregistered [ 7490.993394] Key type ._llcrypt registered [ 7490.995136] Key type .llcrypt registered [ 7491.103659] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7507.881589] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 7521.357940] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7522.053311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7522.157106] alg: No test for adler32 (adler32-zlib) [ 7523.164566] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 7523.369396] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 7525.007177] Key type lgssc registered [ 7525.986656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7533.412033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7539.150582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7544.626962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7551.768367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7557.944838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7557.989443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7557.998589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7559.230512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7559.281786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7559.369879] Lustre: lustre-MDT0000: new disk, initializing [ 7559.465696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7559.482886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7563.132763] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7573.180495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7573.271995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7573.373974] Lustre: 291714:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7573.461255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7573.465097] Lustre: Skipped 1 previous similar message [ 7573.518840] Lustre: lustre-MDT0001: new disk, initializing [ 7573.618680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7573.668714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7573.682558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7577.171554] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7583.569386] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7586.567556] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7592.020673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7592.104726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7592.304250] Lustre: lustre-OST0000: new disk, initializing [ 7592.307398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7592.376215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7597.734725] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7600.716194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7600.731764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7600.920192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7604.824208] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7608.062805] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7608.230347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7611.797879] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7611.988496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7616.479790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7616.485410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7616.497313] LustreError: Skipped 1 previous similar message [ 7616.503853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7619.768100] LustreError: 294653:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7619.914740] Lustre: server umount lustre-OST0000 complete [ 7624.672867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7624.676717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7624.682420] Lustre: Skipped 1 previous similar message [ 7624.686516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7624.689296] Lustre: Skipped 1 previous similar message [ 7626.724994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7626.728603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7629.807927] LustreError: 294855:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7629.812232] LustreError: 294855:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7629.882661] Lustre: server umount lustre-MDT0000 complete [ 7633.092549] LustreError: 291707:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340624 with bad export cookie 6569234058696369618 [ 7633.094097] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7633.122167] LustreError: 291707:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7633.310888] LustreError: 295058:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7633.319296] LustreError: 295058:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7633.498813] Lustre: server umount lustre-MDT0001 complete [ 7641.031191] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:50:31 (1763340631) [ 7646.397436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7646.736474] LustreError: 295729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7646.750483] LustreError: 295729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7646.798569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7649.529866] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7651.819027] LustreError: 295729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7656.113837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7656.507829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7660.058871] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7664.808891] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7668.547980] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7674.526342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7674.739524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7679.246054] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7684.432634] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7701.472099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7701.476788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7701.490735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7701.499393] Lustre: Skipped 1 previous similar message [ 7703.524812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7706.497480] LustreError: 297998:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7706.509109] LustreError: 297998:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7706.605960] Lustre: server umount lustre-OST0000 complete [ 7712.737327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7712.744097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7712.753174] Lustre: Skipped 1 previous similar message [ 7712.761685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7712.764031] Lustre: Skipped 2 previous similar messages [ 7715.548383] LustreError: 298199:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7715.550980] LustreError: 298199:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7715.640725] Lustre: server umount lustre-MDT0000 complete [ 7718.497667] LustreError: 295710:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340710 with bad export cookie 6569234058696370500 [ 7718.510525] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7718.517306] LustreError: 295710:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7718.756739] Lustre: server umount lustre-MDT0001 complete [ 7727.175933] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 7729.003739] Key type lgssc unregistered [ 7729.204574] LNet: 298934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7729.213070] LNetError: 298934:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7729.226854] LNet: Removed LNI 192.168.203.130@tcp [ 7729.789898] Key type .llcrypt unregistered [ 7729.793523] Key type ._llcrypt unregistered [ 7749.040382] Key type ._llcrypt registered [ 7749.044562] Key type .llcrypt registered [ 7749.105191] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 7762.073459] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7762.711948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7762.776837] alg: No test for adler32 (adler32-zlib) [ 7763.781443] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 7763.941295] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 7765.599406] Key type lgssc registered [ 7766.549386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7773.356710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7778.715790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7784.261378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7789.959259] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7794.994535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7799.526609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7805.286491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7805.345808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7805.361414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7806.550964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7806.579519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7806.636709] Lustre: lustre-MDT0000: new disk, initializing [ 7806.729841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7806.753929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7809.657698] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7818.748349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7818.831816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7818.956144] Lustre: 302733:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7819.017892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7819.020502] Lustre: Skipped 1 previous similar message [ 7819.124966] Lustre: lustre-MDT0001: new disk, initializing [ 7819.241269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7819.286217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7819.304628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7822.586395] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7828.565394] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7830.970818] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7836.471165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7836.568278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7836.877463] Lustre: lustre-OST2710: new disk, initializing [ 7836.894520] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7837.016837] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7841.691120] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7843.376948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7843.392070] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7843.486421] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7847.460349] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7852.776780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7852.846635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7852.946354] Lustre: lustre-OST03e8: new disk, initializing [ 7852.949625] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7853.030835] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7858.001155] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7862.841827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7862.853840] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7862.897574] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7863.255181] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7870.263425] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7870.475152] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7873.937992] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7874.126340] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7877.666144] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7877.868643] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7881.517787] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7881.681969] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7885.792096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7885.796415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7885.808244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7888.364916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7888.369801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7888.402156] Lustre: Skipped 2 previous similar messages [ 7888.424833] Lustre: Skipped 3 previous similar messages [ 7890.964491] LustreError: 307076:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7891.044737] Lustre: server umount lustre-MDT0000 complete [ 7893.809776] LustreError: 302724:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340885 with bad export cookie 8487079353394248206 [ 7893.815934] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7893.821487] LustreError: 302724:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7893.943402] LustreError: 307277:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7893.947337] LustreError: 307277:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7894.062806] Lustre: server umount lustre-MDT0001 complete [ 7906.825592] LustreError: 307477:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7906.828550] LustreError: 307477:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7906.903573] Lustre: server umount lustre-OST2710 complete [ 7920.150753] LustreError: 307681:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7920.153524] LustreError: 307681:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7920.294758] Lustre: server umount lustre-OST03e8 complete [ 7934.229265] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 7941.653974] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7949.234085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7954.050167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7958.170100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7964.559537] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7970.333416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7970.373749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7970.606962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7970.649440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7970.736563] Lustre: lustre-MDT0000: new disk, initializing [ 7970.814703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7970.870755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7974.256748] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7984.315361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7984.370613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7984.454372] Lustre: 310477:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7984.550710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7984.556877] Lustre: Skipped 1 previous similar message [ 7984.693296] Lustre: lustre-MDT0001: new disk, initializing [ 7984.861499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7984.931890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7984.950236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7988.692658] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7995.054310] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7998.637296] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8004.497248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8004.567863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8004.755438] Lustre: lustre-OST0000: new disk, initializing [ 8004.758375] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8004.844921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8009.938864] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8013.889668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8013.906336] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8013.983702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8016.446338] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8019.991265] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8020.191670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8023.350600] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8023.552703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8029.667542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8029.680162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8029.711056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8031.380123] LustreError: 313412:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8031.392800] LustreError: 313412:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8031.606688] Lustre: server umount lustre-OST0000 complete [ 8036.323156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8036.332148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8036.358866] Lustre: Skipped 1 previous similar message [ 8036.365223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8036.367661] Lustre: Skipped 1 previous similar message [ 8041.644914] LustreError: 313614:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8041.657569] LustreError: 313614:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8041.856748] Lustre: server umount lustre-MDT0000 complete [ 8044.946161] LustreError: 310469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341036 with bad export cookie 8487079353394249886 [ 8044.951726] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8044.963846] LustreError: 310469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8045.288417] Lustre: server umount lustre-MDT0001 complete [ 8052.102651] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:57:22 (1763341042) [ 8053.256665] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8054.644587] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:57:24 (1763341044) [ 8057.269780] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 8063.220164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8066.351530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8069.112470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8071.906471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8072.755434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8078.060279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8078.261403] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8078.284877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8078.424067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8081.162893] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8086.948083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8087.068085] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8087.101532] Lustre: 315948:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8090.365176] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8094.609063] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8097.204172] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8102.120217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8102.213070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8102.340046] Lustre: Denying initial registration attempt from nid 192.168.203.130@tcp, specified as failover [ 8102.347279] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8102.352177] LustreError: 317564:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 8102.355906] LustreError: 317564:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 8102.359063] LustreError: 317564:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 8102.411460] Lustre: server umount lustre-OST0000 complete [ 8102.413556] LustreError: 317564:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8108.000129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8108.004068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8108.014974] Lustre: Skipped 1 previous similar message [ 8108.019469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8108.022138] Lustre: Skipped 2 previous similar messages [ 8113.458143] LustreError: 317932:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 8113.461456] LustreError: 317932:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8113.510550] Lustre: server umount lustre-MDT0000 complete [ 8114.656105] LustreError: 315960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8114.668855] LustreError: 315960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 8115.838063] LustreError: 315941:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341107 with bad export cookie 8487079353394250768 [ 8115.841741] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8115.843413] LustreError: 315941:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8116.058043] Lustre: server umount lustre-MDT0001 complete [ 8123.267870] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 8125.581805] Key type lgssc unregistered [ 8125.831232] LNet: 318666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8125.841749] LNetError: 318666:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8126.894204] LNet: Removed LNI 192.168.203.130@tcp [ 8127.583247] Key type .llcrypt unregistered [ 8127.590456] Key type ._llcrypt unregistered [ 8136.993178] Key type ._llcrypt registered [ 8136.995214] Key type .llcrypt registered [ 8137.085736] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 8137.875837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8137.898957] alg: No test for adler32 (adler32-zlib) [ 8138.893779] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 8139.063041] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 8140.708238] Key type lgssc registered [ 8141.554085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8147.525321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8150.567644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8153.284730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8156.314917] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8157.176757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8169.440860] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 8175.996965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8176.019291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8177.220758] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8177.240699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8177.405211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8180.713673] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8188.291356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8188.393770] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8188.446452] Lustre: 321403:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8188.642092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8191.871507] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8195.672286] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8198.618945] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8204.473296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8204.548986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8204.702044] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8204.834423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8208.672969] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8212.002970] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8213.458188] LustreError: 323677:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8213.563109] Lustre: server umount lustre-OST0000 complete [ 8219.617244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8219.621844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8219.640572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8220.643868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8220.653353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8223.337326] LustreError: 323886:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 8223.343398] LustreError: 323886:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8223.405448] Lustre: server umount lustre-MDT0000 complete [ 8226.010066] LustreError: 321396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341217 with bad export cookie 5793564292793617163 [ 8226.019903] LustreError: 321396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8226.021454] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8226.127713] LustreError: 324088:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8226.130544] LustreError: 324088:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8226.244249] Lustre: server umount lustre-MDT0001 complete [ 8233.419549] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 8235.588448] Key type lgssc unregistered [ 8235.791676] LNet: 324621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8235.805899] LNetError: 324621:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8236.838143] LNet: Removed LNI 192.168.203.130@tcp [ 8237.451544] Key type .llcrypt unregistered [ 8237.461081] Key type ._llcrypt unregistered [ 8256.434845] Key type ._llcrypt registered [ 8256.436142] Key type .llcrypt registered [ 8256.536398] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 8257.151689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8257.171798] alg: No test for adler32 (adler32-zlib) [ 8258.196901] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 8258.359370] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 8260.048418] Key type lgssc registered [ 8261.001726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8266.929593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8266.961101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8268.360301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8271.801598] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8278.498068] LustreError: 325998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8278.518354] LustreError: 325998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8279.249776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8279.540144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8282.893937] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8287.291847] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8290.360273] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8295.679991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8295.914770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8299.851814] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8303.686984] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8320.480495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8320.488677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8320.505099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8324.579322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8324.579629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8324.589083] Lustre: Skipped 1 previous similar message [ 8326.444827] LustreError: 328281:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8326.550962] Lustre: server umount lustre-MDT0000 complete [ 8329.700897] LustreError: 325997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8329.751613] LustreError: 326728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341321 with bad export cookie 17639859123708615643 [ 8329.758287] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8329.986457] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.30@tcp (stopping) [ 8329.989895] Lustre: Skipped 3 previous similar messages [ 8334.817227] LustreError: 325993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8334.817514] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8334.834780] LustreError: 325993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8334.841431] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8339.936211] LustreError: 325993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8339.942367] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8339.946573] LustreError: 325993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8339.951274] Lustre: Skipped 1 previous similar message [ 8344.031163] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8344.126659] LustreError: 328483:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8344.131120] LustreError: 328483:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8344.257486] Lustre: server umount lustre-MDT0001 complete [ 8349.144408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8356.025301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8356.327487] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf4cd5e69bc95fa74 [ 8356.340976] Lustre: MGC192.168.203.130@tcp: Connection restored to 0@lo (at 0@lo) [ 8356.588424] LustreError: 327617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8356.594555] LustreError: 327617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8356.660482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8356.718381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 8360.036443] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8367.084671] LustreError: 327617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8367.102378] LustreError: 327617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8367.678954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8367.951680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8371.075154] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8371.258847] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8375.772314] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8377.116475] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8380.523207] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8382.084977] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8386.530063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8386.532915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8386.535721] Lustre: Skipped 1 previous similar message [ 8386.545279] Lustre: Skipped 2 previous similar messages [ 8392.406158] LustreError: 330788:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8392.409268] LustreError: 330788:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8392.513744] Lustre: server umount lustre-OST0000 complete [ 8396.770583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8402.141149] LustreError: 330989:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8402.153264] LustreError: 330989:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8402.232252] Lustre: server umount lustre-MDT0000 complete [ 8405.571809] LustreError: 325979:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341397 with bad export cookie 17639859123708623476 [ 8405.579038] LustreError: 325979:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8405.579889] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8405.902634] Lustre: server umount lustre-MDT0001 complete [ 8415.412520] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 8418.278488] Key type lgssc unregistered [ 8418.596426] LNet: 331724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8418.603950] LNetError: 331724:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8418.621614] LNet: Removed LNI 192.168.203.130@tcp [ 8419.420711] Key type .llcrypt unregistered [ 8419.422477] Key type ._llcrypt unregistered [ 8441.291169] Key type ._llcrypt registered [ 8441.292625] Key type .llcrypt registered [ 8441.410698] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 8442.294414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8442.356815] alg: No test for adler32 (adler32-zlib) [ 8443.383034] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 8443.548176] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 8445.215186] Key type lgssc registered [ 8446.126150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8452.774877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8452.786172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8454.106749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8456.846938] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8463.681098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8464.113361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8467.645310] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8473.108869] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8476.346763] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8479.713490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8479.714203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8479.722274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8479.747351] Lustre: Skipped 1 previous similar message [ 8484.832945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8484.836838] Lustre: Skipped 2 previous similar messages [ 8485.168568] LustreError: 334509:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8485.280557] Lustre: server umount lustre-MDT0000 complete [ 8488.748722] LustreError: 333098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341480 with bad export cookie 17230553041435208231 [ 8488.752972] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8488.890755] LustreError: 334712:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8488.893373] LustreError: 334712:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8489.080627] Lustre: server umount lustre-MDT0001 complete [ 8503.065692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8503.409306] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8503.432419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8503.579241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8507.302590] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8514.702419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8514.796271] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8514.829480] Lustre: 335482:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8515.183172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8519.134629] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8524.346730] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8527.786429] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8534.329498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8534.439431] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8534.445981] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8534.454856] LustreError: 337139:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 8534.463214] LustreError: 337139:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 8534.466275] LustreError: 337139:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 8534.510883] Lustre: server umount lustre-OST0000 complete [ 8534.513492] LustreError: 337139:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8541.877380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8542.066873] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8542.242505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8546.859963] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8549.508443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 8552.259717] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8559.687357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8559.744588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8559.782197] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8559.855525] Lustre: lustre-OST0001: new disk, initializing [ 8559.865300] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8559.953361] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8565.142521] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8569.427251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8569.433901] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8569.522938] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8569.694957] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8574.947376] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8574.952630] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8574.961928] LustreError: Skipped 1 previous similar message [ 8574.962779] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8574.988633] Lustre: Skipped 1 previous similar message [ 8578.310285] LustreError: 339261:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 8578.323386] LustreError: 339261:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8578.408143] Lustre: server umount lustre-OST0001 complete [ 8580.085524] LustreError: 338084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8580.092635] LustreError: 338084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8585.186157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8585.187192] LustreError: 338084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8585.189800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8585.198767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8585.198777] Lustre: Skipped 1 previous similar message [ 8585.210646] LustreError: 338084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8585.228627] Lustre: Skipped 1 previous similar message [ 8587.513188] LustreError: 339464:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8587.518019] LustreError: 339464:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8587.610666] Lustre: server umount lustre-OST0000 complete [ 8591.841601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8591.845130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8591.853655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8591.856360] Lustre: Skipped 1 previous similar message [ 8597.286651] LustreError: 339666:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8597.297825] LustreError: 339666:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8597.425289] Lustre: server umount lustre-MDT0000 complete [ 8600.549825] LustreError: 339263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8600.575429] LustreError: 339263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8601.074242] LustreError: 339667:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341592 with bad export cookie 17230553041435208784 [ 8601.083128] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8601.094055] LustreError: 339667:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8601.524308] Lustre: server umount lustre-MDT0001 complete [ 8611.627101] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 8614.524976] Key type lgssc unregistered [ 8614.822554] LNet: 340399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8614.830735] LNetError: 340399:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8614.872304] LNet: Removed LNI 192.168.203.130@tcp [ 8615.598872] Key type .llcrypt unregistered [ 8615.601923] Key type ._llcrypt unregistered [ 8644.651121] Key type ._llcrypt registered [ 8644.652731] Key type .llcrypt registered [ 8644.705488] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 8658.707390] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 8660.179704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8660.207629] alg: No test for adler32 (adler32-zlib) [ 8661.218408] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 8661.453411] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 8663.111772] Key type lgssc registered [ 8664.289628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8672.415100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8678.161637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8683.759639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8689.477630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8695.583244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8695.637082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8695.660304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8696.860510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8696.894435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8697.004859] Lustre: lustre-MDT0000: new disk, initializing [ 8697.117389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8697.140542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8700.262876] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8708.887499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8708.953419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8709.012670] Lustre: 344348:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8709.056991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8709.060675] Lustre: Skipped 1 previous similar message [ 8709.107925] Lustre: lustre-MDT0001: new disk, initializing [ 8709.190288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8709.241460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8709.249088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8712.072495] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8718.218098] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8721.331664] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8726.553935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8726.603597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8726.790168] Lustre: lustre-OST0000: new disk, initializing [ 8726.793808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8726.899791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8732.063837] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8736.840498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8736.856170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8736.964639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8739.136978] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8742.884781] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8743.038575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8746.379920] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8746.568586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8752.612193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8752.612398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8752.615662] LustreError: Skipped 1 previous similar message [ 8752.624806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8752.642413] Lustre: Skipped 1 previous similar message [ 8754.358108] LustreError: 347287:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8754.472935] Lustre: server umount lustre-OST0000 complete [ 8757.730044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8757.731974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8757.742106] Lustre: Skipped 2 previous similar messages [ 8757.747282] Lustre: Skipped 1 previous similar message [ 8762.852044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8762.856476] Lustre: Skipped 1 previous similar message [ 8763.545070] LustreError: 347488:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8763.548114] LustreError: 347488:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8763.646853] Lustre: server umount lustre-MDT0000 complete [ 8766.967033] LustreError: 344341:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341758 with bad export cookie 5554145861686533566 [ 8766.969323] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8766.994052] LustreError: 344341:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8767.183904] LustreError: 347691:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8767.187182] LustreError: 347691:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8767.367786] Lustre: server umount lustre-MDT0001 complete [ 8775.391165] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 20:09:25 (1763341765) [ 8780.362043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8785.117956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8805.616445] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 8813.631544] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 8822.479957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8828.185854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8834.133232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8839.866526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8846.161142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8846.221690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8846.434945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8846.469355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8846.554679] Lustre: lustre-MDT0000: new disk, initializing [ 8846.674151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8846.689654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8850.802651] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8860.678856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8860.747091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8860.832130] Lustre: 351469:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8860.894846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8860.902859] Lustre: Skipped 1 previous similar message [ 8861.008409] Lustre: lustre-MDT0001: new disk, initializing [ 8861.106097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8861.150246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8861.169148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8864.709219] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8871.327312] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8875.007700] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8882.484548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8882.595915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8882.958121] Lustre: lustre-OST0000: new disk, initializing [ 8882.969461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8883.082160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8888.938917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8888.953945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8889.123734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8889.669439] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8897.209097] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8901.000403] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8901.177405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8904.491972] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8904.690624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8909.792136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8909.804302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8909.805665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8909.817161] Lustre: Skipped 1 previous similar message [ 8909.838322] Lustre: Skipped 1 previous similar message [ 8912.446660] LustreError: 354410:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8912.448785] LustreError: 354410:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8912.582436] Lustre: server umount lustre-OST0000 complete [ 8917.472921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8917.476881] LustreError: Skipped 1 previous similar message [ 8917.479752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8917.505065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8921.853092] LustreError: 354611:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8921.856575] LustreError: 354611:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8921.942722] Lustre: server umount lustre-MDT0000 complete [ 8925.356650] LustreError: 351460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341916 with bad export cookie 5554145861686534448 [ 8925.363996] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8925.365868] LustreError: 351460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8925.717379] Lustre: server umount lustre-MDT0001 complete [ 8934.852296] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 20:12:04 (1763341924) [ 8942.236143] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 20:12:12 (1763341932) [ 8950.085873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8950.708865] LustreError: 355829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8950.726801] LustreError: 355829:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8950.810298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8954.254644] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8955.876893] LustreError: 355829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8960.997527] LustreError: 355829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8961.786770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8965.625424] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8969.584177] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8972.351597] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8977.496586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8982.284075] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8986.496196] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9001.435745] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9002.682348] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9003.881358] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9005.120432] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9006.379865] Lustre: Failing over lustre-MDT0000 [ 9006.508272] LustreError: 358285:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9006.511390] LustreError: 358285:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9006.564741] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9006.571928] Lustre: Skipped 1 previous similar message [ 9006.579111] LustreError: 355823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9006.586138] LustreError: 355823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9006.593488] Lustre: server umount lustre-MDT0000 complete [ 9011.683268] LustreError: 355824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9011.696181] LustreError: 355824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9021.921479] LustreError: 355829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9021.944562] LustreError: 355829:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9021.959786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9022.069642] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9022.242467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9022.247166] Lustre: Skipped 2 previous similar messages [ 9022.311220] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9022.682475] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9025.386900] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9027.558189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9027.593860] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9027.652361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9031.647675] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9032.810820] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9035.696184] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9037.289718] Lustre: Failing over lustre-MDT0000 [ 9037.451607] LustreError: 359366:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9037.459293] LustreError: 359366:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9037.504714] Lustre: server umount lustre-MDT0000 complete [ 9037.795361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9037.798108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9037.824032] Lustre: Skipped 4 previous similar messages [ 9038.082408] LustreError: 357984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9038.100880] LustreError: 357984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9054.910627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9054.988554] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9055.240739] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9058.686199] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9059.545698] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9060.326143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9060.329268] Lustre: Skipped 2 previous similar messages [ 9060.339077] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9060.368431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9064.850411] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9065.980290] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9070.562044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9070.565605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9070.587293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9070.594894] Lustre: Skipped 4 previous similar messages [ 9076.754752] LustreError: 360399:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9076.773756] LustreError: 360399:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9077.048606] Lustre: server umount lustre-MDT0000 complete [ 9080.456064] LustreError: 355810:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342071 with bad export cookie 5554145861686537143 [ 9080.457644] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9080.475489] LustreError: 355810:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9080.802377] Lustre: server umount lustre-MDT0001 complete [ 9095.126559] Lustre: server umount lustre-OST0000 complete [ 9107.616263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9107.884268] LustreError: 361630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9107.909024] LustreError: 361630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 9107.951170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9107.956191] Lustre: Skipped 1 previous similar message [ 9110.820217] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9117.600779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9120.742240] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9125.201834] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9128.843883] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9134.921474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9141.092872] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9145.486275] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9153.776880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9163.233132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9163.238644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9163.246563] Lustre: Skipped 2 previous similar messages [ 9163.252317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9163.258343] Lustre: Skipped 3 previous similar messages [ 9165.623483] LustreError: 363898:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9165.626565] LustreError: 363898:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 9165.714273] Lustre: server umount lustre-OST0000 complete [ 9183.199152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9183.380219] Lustre: server umount lustre-MDT0000 complete [ 9184.736251] LustreError: 361630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9184.746476] LustreError: 361630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9186.537805] LustreError: 361610:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342178 with bad export cookie 5554145861686537843 [ 9186.539120] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9186.551822] LustreError: 361610:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9186.823422] Lustre: server umount lustre-MDT0001 complete [ 9193.681179] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 9195.877565] Key type lgssc unregistered [ 9196.083187] LNet: 364836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9196.088692] LNetError: 364836:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9197.159451] LNet: Removed LNI 192.168.203.130@tcp [ 9197.864659] Key type .llcrypt unregistered [ 9197.866356] Key type ._llcrypt unregistered [ 9221.940351] Key type ._llcrypt registered [ 9221.942195] Key type .llcrypt registered [ 9222.017578] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 9223.055943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9223.111054] alg: No test for adler32 (adler32-zlib) [ 9224.153908] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 9224.327209] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 9225.975496] Key type lgssc registered [ 9227.028639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9233.725194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9233.755134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9235.153454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9238.266256] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9244.644320] LustreError: 366341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9244.660386] LustreError: 366341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9245.463351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9245.719668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9248.922984] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9253.826588] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9257.365237] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9264.343867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9264.635542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9269.652435] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9273.990599] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9279.990401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9293.919372] LustreError: 368608:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9294.003602] Lustre: server umount lustre-MDT0000 complete [ 9295.330996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9295.332927] LustreError: 368508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9295.355717] Lustre: Skipped 2 previous similar messages [ 9295.365555] LustreError: 368508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9297.235921] LustreError: 366321:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342288 with bad export cookie 9308007656424183099 [ 9297.245581] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9297.422987] LustreError: 368809:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9297.438032] LustreError: 368809:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9297.644302] Lustre: server umount lustre-MDT0001 complete [ 9310.975767] LustreError: 369011:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9310.982577] LustreError: 369011:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9311.091585] Lustre: server umount lustre-OST0000 complete [ 9321.817334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9322.292708] LustreError: 369741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9322.368929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9326.234624] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9327.596190] LustreError: 369741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9331.687302] LustreError: 369741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9334.283884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9334.603749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9337.999729] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9342.812671] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9345.867333] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9351.920557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9352.223928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9357.123677] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9361.824915] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9366.578127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9380.834244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9380.839737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9380.850061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9381.859026] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9381.859979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9383.147589] LustreError: 372094:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9383.151407] LustreError: 372094:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9383.212633] Lustre: server umount lustre-MDT0000 complete [ 9386.008512] LustreError: 369720:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342377 with bad export cookie 9308007656424184219 [ 9386.010106] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9386.021990] LustreError: 369720:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9386.248931] Lustre: server umount lustre-MDT0001 complete [ 9399.236917] LustreError: 372496:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9399.244798] LustreError: 372496:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 9399.344961] Lustre: server umount lustre-OST0000 complete [ 9410.286738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9410.672886] LustreError: 373325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9410.739813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9414.017463] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9422.059574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9425.791873] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9430.598424] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9433.608936] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9439.175803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9444.140451] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9449.220191] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9455.920818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9466.340554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9466.343723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9466.347870] Lustre: Skipped 1 previous similar message [ 9466.348717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9466.348723] Lustre: Skipped 2 previous similar messages [ 9469.501070] LustreError: 375589:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9469.505799] LustreError: 375589:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9469.602835] Lustre: server umount lustre-OST0000 complete [ 9473.505841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9473.509574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9473.515453] Lustre: Skipped 1 previous similar message [ 9473.517556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9473.520869] Lustre: Skipped 1 previous similar message [ 9478.529955] Lustre: server umount lustre-MDT0000 complete [ 9481.696986] LustreError: 374698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9481.711461] LustreError: 374698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9481.895866] LustreError: 374942:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342473 with bad export cookie 9308007656424185234 [ 9481.899162] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9481.902769] LustreError: 374942:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9482.271141] Lustre: server umount lustre-MDT0001 complete [ 9490.901984] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 9493.089766] Key type lgssc unregistered [ 9493.421571] LNet: 376526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9493.430268] LNetError: 376526:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9493.461439] LNet: Removed LNI 192.168.203.130@tcp [ 9494.199740] Key type .llcrypt unregistered [ 9494.201490] Key type ._llcrypt unregistered [ 9514.127441] Key type ._llcrypt registered [ 9514.129284] Key type .llcrypt registered [ 9514.221471] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 9515.009834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9515.027202] alg: No test for adler32 (adler32-zlib) [ 9516.033663] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 9516.265336] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 9517.976212] Key type lgssc registered [ 9518.841276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9525.628385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9525.648791] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9525.669560] LustreError: 377925:0:(osd_handler.c:8577:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9525.697722] LustreError: 377925:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9525.702572] LustreError: 377925:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9525.713194] LustreError: 377925:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9525.720657] LustreError: 377925:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9532.665787] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9532.680369] LustreError: 378323:0:(osd_handler.c:8577:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9532.702216] LustreError: 378323:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9532.712600] LustreError: 378323:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9532.717608] LustreError: 378323:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9532.723964] LustreError: 378323:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9549.135713] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 9551.559253] Key type lgssc unregistered [ 9551.797529] LNet: 379309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9551.802701] LNetError: 379309:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9551.833982] LNet: Removed LNI 192.168.203.130@tcp [ 9552.567960] Key type .llcrypt unregistered [ 9552.569724] Key type ._llcrypt unregistered [ 9567.947054] Key type ._llcrypt registered [ 9567.948750] Key type .llcrypt registered [ 9568.030071] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 9580.871489] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 9581.875114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9581.893389] alg: No test for adler32 (adler32-zlib) [ 9582.951331] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 9583.189213] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 9584.847210] Key type lgssc registered [ 9585.691177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9593.242301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9598.890049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9605.011199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9611.345538] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9617.636462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9617.708592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9617.731033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9619.035854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9619.070674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9619.153491] Lustre: lustre-MDT0000: new disk, initializing [ 9619.262293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9619.272522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9623.147549] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9633.696639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9633.781426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9633.864806] Lustre: 382435:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9633.923564] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9633.929596] Lustre: Skipped 1 previous similar message [ 9634.026927] Lustre: lustre-MDT0001: new disk, initializing [ 9634.156819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9634.208831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9634.227452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9638.190426] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9645.405766] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9649.243900] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9656.091564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9656.174293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9656.453626] Lustre: lustre-OST0000: new disk, initializing [ 9656.462419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9656.565689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9662.459072] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9665.632736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9665.656867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9665.763363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9669.565819] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9673.362755] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9673.532635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9677.020549] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9677.195728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9681.377187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9681.379701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9681.382354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9681.397192] Lustre: Skipped 1 previous similar message [ 9686.501158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9686.503992] Lustre: Skipped 2 previous similar messages [ 9691.618821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9693.152845] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9693.193245] LustreError: 385375:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9693.348310] Lustre: server umount lustre-OST0000 complete [ 9696.736461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9696.752755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9696.755775] Lustre: Skipped 2 previous similar messages [ 9701.856802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9701.865410] Lustre: Skipped 1 previous similar message [ 9702.804731] LustreError: 385577:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9702.811523] LustreError: 385577:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9702.888422] Lustre: server umount lustre-MDT0000 complete [ 9706.206218] LustreError: 384182:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342697 with bad export cookie 11816551875188160086 [ 9706.212828] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9706.217171] LustreError: 384182:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9706.335996] LustreError: 385781:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9706.339997] LustreError: 385781:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9706.525240] Lustre: server umount lustre-MDT0001 complete [ 9714.059259] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9715.727902] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 20:25:05 (1763342705) [ 9721.088504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9721.433580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9724.618315] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9726.432558] LustreError: 386500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9726.459738] LustreError: 386500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9730.535511] LustreError: 386500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9732.367035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9732.591322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9736.612734] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9741.161726] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9745.046748] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9751.869871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9752.235727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9757.467175] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9762.525210] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9768.613166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9768.697742] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9768.805261] Lustre: lustre-OST0001: new disk, initializing [ 9768.807744] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9768.885282] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9773.906827] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9774.647767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9774.662026] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9774.737888] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9780.624839] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9784.814825] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9784.823308] Lustre: Skipped 1 previous similar message [ 9784.829038] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9784.832422] Lustre: Skipped 1 previous similar message [ 9790.074014] LustreError: 389700:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 9790.077026] LustreError: 389700:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9790.146120] Lustre: server umount lustre-OST0001 complete [ 9792.727440] LustreError: 388097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9795.047449] LustreError: 388098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9795.066299] LustreError: 388098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9800.172439] LustreError: 388099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9800.184856] LustreError: 388099:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9806.820632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9806.824761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9806.844181] Lustre: Skipped 1 previous similar message [ 9806.872683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9806.876211] Lustre: Skipped 3 previous similar messages [ 9809.895671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9810.405735] LustreError: 389395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9810.420774] LustreError: 389395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9820.127200] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9820.178279] LustreError: 389923:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9820.181448] LustreError: 389923:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9820.332529] Lustre: server umount lustre-OST0000 complete [ 9824.738703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9824.742648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9824.751015] Lustre: Skipped 1 previous similar message [ 9830.091510] LustreError: 390125:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 9830.095186] LustreError: 390125:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9830.164144] Lustre: server umount lustre-MDT0000 complete [ 9830.893393] LustreError: 387086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9830.909930] LustreError: 387086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9834.054492] LustreError: 386480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342825 with bad export cookie 11816551875188160968 [ 9834.055909] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9834.074684] LustreError: 386480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9840.524423] Lustre: server umount lustre-MDT0001 complete [ 9850.373680] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 9852.982617] Key type lgssc unregistered [ 9853.336521] LNet: 390859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9853.347334] LNetError: 390859:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9853.364689] LNet: Removed LNI 192.168.203.130@tcp [ 9854.217888] Key type .llcrypt unregistered [ 9854.222423] Key type ._llcrypt unregistered [ 9884.851940] Key type ._llcrypt registered [ 9884.854150] Key type .llcrypt registered [ 9884.928394] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 9897.961756] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 9898.761105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9898.845494] alg: No test for adler32 (adler32-zlib) [ 9899.883570] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 9900.074196] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 9901.743186] Key type lgssc registered [ 9902.717951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9910.097510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9915.322207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9921.110932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9926.704150] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9932.202885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9932.263440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9932.288798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9933.578929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9933.615891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9933.762211] Lustre: lustre-MDT0000: new disk, initializing [ 9933.873275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9933.906048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9937.692642] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9945.907794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9945.990643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9946.065894] Lustre: 394809:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9946.144309] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9946.147278] Lustre: Skipped 1 previous similar message [ 9946.229410] Lustre: lustre-MDT0001: new disk, initializing [ 9946.319477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9946.363728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9946.375308] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9949.741451] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9956.351925] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9959.989844] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9966.276175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9966.367587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9966.615424] Lustre: lustre-OST0000: new disk, initializing [ 9966.620359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9966.695555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9971.347493] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9974.904971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9974.914391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9974.996592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9977.455439] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9980.883350] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9981.075601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9984.595299] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9984.764770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9990.628146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9990.628861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9990.630150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9990.631837] LustreError: Skipped 1 previous similar message [ 9990.637258] Lustre: Skipped 1 previous similar message [ 9995.747058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9995.751699] Lustre: Skipped 2 previous similar messages [10000.871840] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10000.875333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10000.897127] Lustre: Skipped 1 previous similar message [10000.976144] LustreError: 397748:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10001.130106] Lustre: server umount lustre-OST0000 complete [10005.984533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10005.988175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10006.009033] Lustre: Skipped 1 previous similar message [10006.033442] Lustre: Skipped 1 previous similar message [10011.108365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10011.116320] Lustre: Skipped 1 previous similar message [10019.296769] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10019.385249] LustreError: 397950:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10019.395296] LustreError: 397950:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10019.507954] Lustre: server umount lustre-MDT0000 complete [10022.691696] LustreError: 394800:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343014 with bad export cookie 6345527843647585876 [10022.695335] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10022.706256] LustreError: 394800:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10022.818689] LustreError: 398154:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10022.826259] LustreError: 398154:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10023.000817] Lustre: server umount lustre-MDT0001 complete [10031.326391] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 20:30:21 (1763343021) [10039.727503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10047.157824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10047.559139] Lustre: lustre-MDT0000: new disk, initializing [10047.581823] LustreError: 399422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10047.600686] LustreError: 399422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10047.651527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10051.210658] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10053.092967] LustreError: 399422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10058.211661] LustreError: 399422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10058.896674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10059.208344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10062.624087] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10067.731667] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10071.159413] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10074.598607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10074.604606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10074.614845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10074.617896] Lustre: Skipped 2 previous similar messages [10077.672231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10079.431694] LustreError: 400764:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10079.436949] LustreError: 400764:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10079.533127] Lustre: server umount lustre-MDT0000 complete [10082.784036] LustreError: 399422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10082.796051] LustreError: 399422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10082.858469] LustreError: 399402:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343074 with bad export cookie 6345527843647586758 [10082.860160] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10082.869714] LustreError: 399402:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10083.181878] Lustre: server umount lustre-MDT0001 complete [10091.238258] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 20:31:21 (1763343081) [10097.491912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10097.876428] LustreError: 401688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10097.955294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10101.196851] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10107.400951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10110.813637] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10115.303163] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10118.469369] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10124.584799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10124.870606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10124.875215] Lustre: Skipped 1 previous similar message [10129.668425] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10134.557817] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10152.300773] Lustre: Permanently deactivating lustre-OST0000 [10152.318754] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10161.769969] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10163.732718] LustreError: 404402:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10163.737600] LustreError: 404402:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10163.748492] LustreError: 404402:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10166.754061] LustreError: 404451:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10166.765284] LustreError: 404451:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10174.659295] LustreError: 404548:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10174.675983] LustreError: 404548:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10174.836961] Lustre: server umount lustre-OST0000 complete [10177.262993] LustreError: 404700:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10177.267759] LustreError: 404700:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10179.553817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10179.557141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10179.563970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10179.566907] Lustre: Skipped 2 previous similar messages [10193.376092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10193.461781] LustreError: 404798:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10193.463955] LustreError: 404798:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10193.532167] Lustre: server umount lustre-MDT0000 complete [10195.938782] LustreError: 403845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10195.947345] LustreError: 403845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10196.821385] LustreError: 404058:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343188 with bad export cookie 6345527843647587311 [10196.827111] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10196.828870] LustreError: 404058:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [10207.376721] Lustre: server umount lustre-MDT0001 complete [10214.459854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10217.673169] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10219.970545] LustreError: 405845:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10225.139403] Lustre: 406184:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.130@tcp [10226.713309] LustreError: 406281:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10226.720085] LustreError: 406281:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10226.794966] Lustre: server umount MGS complete [10236.882522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10237.246547] LustreError: 406910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10237.268760] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10237.301694] Lustre: Skipped 1 previous similar message [10237.383488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10241.287400] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10249.877461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10250.281461] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10254.213278] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10259.769607] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10263.951413] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10265.766918] Lustre: Permanently reactivating lustre-OST0000 [10265.808085] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10265.811400] Lustre: Skipped 2 previous similar messages [10287.071184] LustreError: 408317:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10294.493543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10300.185095] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10304.915670] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10312.162526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10312.164517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10312.211460] Lustre: Skipped 1 previous similar message [10312.215601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10312.215615] Lustre: Skipped 7 previous similar messages [10315.082102] LustreError: 409885:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10315.086872] LustreError: 409885:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10315.241989] Lustre: server umount lustre-OST0000 complete [10321.890752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10324.825165] Lustre: server umount lustre-MDT0000 complete [10327.526830] LustreError: 406906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10327.539792] LustreError: 406906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10327.653288] LustreError: 406891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343319 with bad export cookie 6345527843647588270 [10327.657207] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10327.667049] LustreError: 406891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10327.911247] Lustre: server umount lustre-MDT0001 complete [10335.619864] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [10337.815150] Key type lgssc unregistered [10338.053272] LNet: 410825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10338.058292] LNetError: 410825:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10338.082358] LNet: Removed LNI 192.168.203.130@tcp [10338.768233] Key type .llcrypt unregistered [10338.773576] Key type ._llcrypt unregistered [10351.904568] Key type ._llcrypt registered [10351.906415] Key type .llcrypt registered [10351.985742] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [10363.758626] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [10364.404732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10364.431989] alg: No test for adler32 (adler32-zlib) [10365.413361] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [10365.623566] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [10367.255184] Key type lgssc registered [10368.131067] Lustre: Echo OBD driver; http://www.lustre.org/ [10376.192589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10382.338689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10387.257971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10392.435825] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10398.885951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10398.959876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10398.968449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10400.271436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10400.306540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10400.384255] Lustre: lustre-MDT0000: new disk, initializing [10400.454130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10400.486596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10403.824944] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10412.871690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10412.929161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10413.035194] Lustre: 413936:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10413.093977] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10413.098522] Lustre: Skipped 1 previous similar message [10413.152801] Lustre: lustre-MDT0001: new disk, initializing [10413.265310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10413.303404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10413.309840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10416.416097] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10422.223532] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10425.380054] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10430.973130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10431.059123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10431.314107] Lustre: lustre-OST0000: new disk, initializing [10431.322121] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10431.399528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10432.455820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10432.461213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10432.583857] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10436.844048] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10443.765856] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10447.275822] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10447.495357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10450.851233] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10451.023315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10452.973862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10452.977794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10453.005979] LustreError: Skipped 1 previous similar message [10453.009679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10458.087086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10458.092729] Lustre: Skipped 1 previous similar message [10463.205872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10467.295150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10467.322182] LustreError: 416873:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10467.433489] Lustre: server umount lustre-OST0000 complete [10473.440851] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10473.449117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10473.450031] Lustre: Skipped 2 previous similar messages [10473.463551] Lustre: Skipped 2 previous similar messages [10476.764882] LustreError: 417079:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10476.767476] LustreError: 417079:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10476.827273] Lustre: server umount lustre-MDT0000 complete [10479.578811] LustreError: 413927:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343471 with bad export cookie 6873631077252860676 [10479.589152] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10479.603417] LustreError: 413927:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10479.714885] LustreError: 417281:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10479.724230] LustreError: 417281:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10479.908951] Lustre: server umount lustre-MDT0001 complete [10487.788425] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10489.233725] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:37:59 (1763343479) [10494.935629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10495.366920] LustreError: 418001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10495.389336] LustreError: 418001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10495.458909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10499.042585] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10500.586313] LustreError: 418001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10504.693047] LustreError: 418001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10506.789071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10507.171597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10510.519494] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10515.294542] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10518.857922] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10524.569762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10524.985482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10530.164875] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10535.026385] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10536.776647] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10543.708223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10550.763569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10550.763954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10550.785808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10554.316093] LustreError: 420439:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10554.321866] LustreError: 420439:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10554.459726] Lustre: server umount lustre-OST0000 complete [10558.432394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10558.438286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10558.452109] Lustre: Skipped 1 previous similar message [10560.998138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10561.001106] Lustre: Skipped 3 previous similar messages [10572.255122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10572.365570] LustreError: 420640:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10572.374505] LustreError: 420640:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10572.480449] Lustre: server umount lustre-MDT0000 complete [10576.186173] LustreError: 417982:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343567 with bad export cookie 6873631077252861558 [10576.191683] LustreError: 417982:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [10576.195818] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10576.572773] Lustre: server umount lustre-MDT0001 complete [10585.496693] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [10588.210532] Key type lgssc unregistered [10588.507839] LNet: 421378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10588.512682] LNetError: 421378:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10588.530536] LNet: Removed LNI 192.168.203.130@tcp [10589.391196] Key type .llcrypt unregistered [10589.399729] Key type ._llcrypt unregistered [10609.034025] Key type ._llcrypt registered [10609.037778] Key type .llcrypt registered [10609.112552] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [10609.415567] Key type .llcrypt unregistered [10609.417121] Key type ._llcrypt unregistered [10623.142980] Key type ._llcrypt registered [10623.144885] Key type .llcrypt registered [10623.290953] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [10624.368481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10624.400819] alg: No test for adler32 (adler32-zlib) [10625.582508] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [10625.819703] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [10627.551234] Key type lgssc registered [10628.898293] Lustre: Echo OBD driver; http://www.lustre.org/ [10637.592964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10637.632945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10639.363928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10644.050772] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10648.692019] LustreError: 423531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10648.724259] LustreError: 423531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10652.390606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10652.661708] LustreError: 423531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10652.682824] LustreError: 423531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10652.700368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10657.766140] LustreError: 423531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10657.795791] LustreError: 423531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10658.675229] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10662.884858] LustreError: 423527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10662.928739] LustreError: 423527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10663.914263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10664.133391] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10669.025368] LustreError: 423526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10669.041872] LustreError: 423526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10670.217396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10670.532664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10674.165733] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10680.810315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10680.810654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10680.826115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10680.833733] Lustre: Skipped 1 previous similar message [10685.920456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10685.940375] Lustre: Skipped 2 previous similar messages [10686.263298] LustreError: 425586:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [10686.373730] Lustre: server umount lustre-OST0000 complete [10689.067193] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10689.079111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10689.103497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10689.182675] LustreError: 425787:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10689.185907] LustreError: 425787:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10689.254688] Lustre: server umount lustre-MDT0000 complete [10691.042370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10691.043717] LustreError: 425141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10691.059888] LustreError: 425141:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10692.246278] LustreError: 423510:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343683 with bad export cookie 13636292572175019502 [10692.252284] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10692.259174] LustreError: 423510:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10692.360649] LustreError: 425987:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10692.364639] LustreError: 425987:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10692.554424] Lustre: server umount lustre-MDT0001 complete [10700.971331] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [10703.079811] Key type lgssc unregistered [10703.324138] LNet: 426519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10703.337206] LNetError: 426519:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10703.355444] LNet: Removed LNI 192.168.203.130@tcp [10704.098197] Key type .llcrypt unregistered [10704.099717] Key type ._llcrypt unregistered [10722.668619] Key type ._llcrypt registered [10722.669961] Key type .llcrypt registered [10722.760070] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [10723.341243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10723.372090] alg: No test for adler32 (adler32-zlib) [10724.395602] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [10724.611306] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [10726.271248] Key type lgssc registered [10727.441341] Lustre: Echo OBD driver; http://www.lustre.org/ [10733.866689] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10733.934966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10749.476841] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10754.591342] LustreError: 427867:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.130@tcp: failed processing log, type 4: rc = -110 [10781.280658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10786.177789] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10790.348876] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10795.688267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10796.007654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10796.080024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10799.057417] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10801.122621] LustreError: 427893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10801.133233] LustreError: 427893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10805.131076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10805.385699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10808.470178] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10812.523840] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10815.462955] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10820.577321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10820.581462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10820.597129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10821.093492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10821.126683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10821.132510] Lustre: Skipped 1 previous similar message [10826.038181] LustreError: 430147:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [10826.116459] Lustre: server umount lustre-OST0000 complete [10828.711281] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10828.715969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10828.727186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10831.328886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10831.330684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10831.354228] Lustre: Skipped 1 previous similar message [10836.449134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10843.103444] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10843.176778] LustreError: 430348:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [10843.187033] LustreError: 430348:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10843.264119] Lustre: server umount lustre-MDT0000 complete [10845.664635] LustreError: 430059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10845.680832] LustreError: 430059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10846.179354] LustreError: 427876:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343837 with bad export cookie 8811562791811271154 [10846.189509] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10846.201068] LustreError: 427876:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10846.293601] LustreError: 430554:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10846.298769] LustreError: 430554:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10846.414769] Lustre: server umount lustre-MDT0001 complete [10855.851938] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [10857.997431] Key type lgssc unregistered [10858.296205] LNet: 431085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10858.312599] LNetError: 431085:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10859.375529] LNet: Removed LNI 192.168.203.130@tcp [10860.068831] Key type .llcrypt unregistered [10860.072718] Key type ._llcrypt unregistered [10881.662374] Key type ._llcrypt registered [10881.664120] Key type .llcrypt registered [10881.778038] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [10882.473774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10882.502978] alg: No test for adler32 (adler32-zlib) [10883.489368] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [10883.636853] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [10885.279170] Key type lgssc registered [10886.186647] Lustre: Echo OBD driver; http://www.lustre.org/ [10892.243134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10892.267532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10893.615394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10896.549161] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10902.287446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10902.575560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10905.955807] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10910.631159] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10913.403521] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10918.374468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10918.614380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10919.654709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10923.368320] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10928.466066] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10933.218437] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10933.219887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10933.223322] Lustre: Skipped 1 previous similar message [10933.238174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10933.257811] Lustre: Skipped 2 previous similar messages [10936.046433] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.30@tcp (stopping) [10938.343350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10938.357824] Lustre: Skipped 2 previous similar messages [10939.285981] LustreError: 434729:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10939.432657] Lustre: server umount lustre-MDT0000 complete [10941.146581] LustreError: 432477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10941.171796] LustreError: 432477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10941.519303] LustreError: 433200:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.30@tcp arrived at 1763343933 with bad export cookie 12111180526300403151 [10943.267633] LustreError: 433200:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343934 with bad export cookie 12111180526300402255 [10943.273030] LustreError: 433200:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10943.277793] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10943.458374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10943.462611] LustreError: 432481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10943.474620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10943.505619] LustreError: 432481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10943.519827] Lustre: Skipped 1 previous similar message [10943.520626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10948.577299] LustreError: 432476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10948.581050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10948.585183] LustreError: 432476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10949.375047] LustreError: 434931:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10949.377859] LustreError: 434931:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10949.527308] Lustre: server umount lustre-OST0000 complete [10952.674097] LustreError: 432481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10952.682860] LustreError: 432481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10955.267184] LustreError: 435285:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10955.271967] LustreError: 435285:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10955.486734] Lustre: server umount lustre-MDT0001 complete [10964.766255] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [10966.965734] Key type lgssc unregistered [10967.189214] LNet: 435817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10967.194939] LNetError: 435817:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10967.212259] LNet: Removed LNI 192.168.203.130@tcp [10967.999990] Key type .llcrypt unregistered [10968.005672] Key type ._llcrypt unregistered [10986.707150] Key type ._llcrypt registered [10986.708803] Key type .llcrypt registered [10986.797805] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [10987.608263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10987.716988] alg: No test for adler32 (adler32-zlib) [10988.763316] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [10988.964560] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [10990.631210] Key type lgssc registered [10991.522623] Lustre: Echo OBD driver; http://www.lustre.org/ [10997.833896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10997.863214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10999.387202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11002.241774] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11008.483908] LustreError: 437193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11008.495032] LustreError: 437193:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11009.144622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11009.416353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11013.116699] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11018.350428] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11021.850849] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11028.369415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11028.855201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11034.326100] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11038.747337] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11044.215557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11049.440937] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11049.468640] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11052.613994] LustreError: 439440:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11052.743327] Lustre: server umount lustre-MDT0001 complete [11054.561255] LustreError: 437194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11054.571654] LustreError: 437194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11059.680445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11059.684326] LustreError: 437194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11059.690629] Lustre: Skipped 1 previous similar message [11059.691892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11059.691899] Lustre: Skipped 2 previous similar messages [11063.427167] LustreError: 439642:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11063.430326] LustreError: 439642:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11063.546680] Lustre: server umount lustre-OST0000 complete [11064.802387] LustreError: 437194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11066.367495] LustreError: 439843:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11066.374568] LustreError: 439843:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11066.525452] Lustre: server umount lustre-MDT0000 complete [11076.701249] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [11078.679268] Key type lgssc unregistered [11078.870312] LNet: 440527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11078.875639] LNetError: 440527:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11078.887549] LNet: Removed LNI 192.168.203.130@tcp [11079.473139] Key type .llcrypt unregistered [11079.475539] Key type ._llcrypt unregistered [11100.261541] Key type ._llcrypt registered [11100.263435] Key type .llcrypt registered [11100.376898] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [11112.422525] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [11113.623246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11113.637082] alg: No test for adler32 (adler32-zlib) [11114.822561] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [11115.065506] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [11116.791241] Key type lgssc registered [11117.962037] Lustre: Echo OBD driver; http://www.lustre.org/ [11125.942900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11132.606390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11138.587786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11144.583571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11153.243933] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [11161.610653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11167.590797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11167.639728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11167.650153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11168.884453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11168.938140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11169.045663] Lustre: lustre-MDT0000: new disk, initializing [11169.186211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11169.203712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11172.712660] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11180.986139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11187.396428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11187.468933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11187.761897] Lustre: lustre-OST0000: new disk, initializing [11187.777412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11187.780356] Lustre: Skipped 1 previous similar message [11187.872885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11193.166491] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11193.976385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11193.986842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11194.037419] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11199.960257] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11207.222662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11213.553665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11213.634435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11213.716687] Lustre: 444483:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11213.770262] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11213.837395] Lustre: lustre-MDT0001: new disk, initializing [11213.947307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11213.999934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11214.021352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11217.671831] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11224.850712] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11225.053233] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11228.182249] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11228.378787] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11231.712716] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11231.717217] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11231.733848] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11233.006446] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.30@tcp (stopping) [11236.322362] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11236.325404] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11237.657860] LustreError: 447902:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11237.719637] Lustre: server umount lustre-MDT0001 complete [11246.569205] LustreError: 444491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11246.583367] LustreError: 444491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11251.685347] LustreError: 444491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11251.706051] LustreError: 444491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11255.264428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11255.269089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11255.280724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11255.283386] Lustre: Skipped 1 previous similar message [11256.800608] LustreError: 444495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11258.430929] LustreError: 448174:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [11258.437815] LustreError: 448174:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11258.563709] Lustre: server umount lustre-OST0000 complete [11267.376979] LustreError: 448376:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [11267.381911] LustreError: 448376:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11267.594065] Lustre: server umount lustre-MDT0000 complete [11277.827386] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [11280.139523] Key type lgssc unregistered [11280.361651] LNet: 449061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11280.366635] LNetError: 449061:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11280.384925] LNet: Removed LNI 192.168.203.130@tcp [11281.202704] Key type .llcrypt unregistered [11281.204463] Key type ._llcrypt unregistered [11313.433622] Key type ._llcrypt registered [11313.435851] Key type .llcrypt registered [11313.527588] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [11313.774273] Key type .llcrypt unregistered [11313.778374] Key type ._llcrypt unregistered [11327.980543] Key type ._llcrypt registered [11327.982029] Key type .llcrypt registered [11328.080909] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [11329.030432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11329.192597] alg: No test for adler32 (adler32-zlib) [11330.216242] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [11330.445695] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [11332.175401] Key type lgssc registered [11333.287142] Lustre: Echo OBD driver; http://www.lustre.org/ [11339.133841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11344.433980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11349.607136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11355.403806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11355.444113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11355.453984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11356.624238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11356.650920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11356.697854] Lustre: lustre-MDT0000: new disk, initializing [11356.753969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11356.766522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11359.876744] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11369.790443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11369.889921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11370.034782] Lustre: 452405:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11370.097651] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11370.100608] Lustre: Skipped 1 previous similar message [11370.174098] Lustre: lustre-MDT0001: new disk, initializing [11370.266617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11370.315246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11370.321765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11374.076545] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11381.385644] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11385.010389] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11391.536275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11391.627962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11391.850918] Lustre: lustre-OST0000: new disk, initializing [11391.854365] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11391.957900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11397.208815] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11399.219826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11399.231526] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11399.308537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11403.485596] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11409.380389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11409.389719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11409.396372] Lustre: Skipped 2 previous similar messages [11409.402548] Lustre: Skipped 1 previous similar message [11414.497394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11414.506607] Lustre: Skipped 2 previous similar messages [11415.251623] LustreError: 454848:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11415.380190] Lustre: server umount lustre-MDT0000 complete [11418.854862] LustreError: 454151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344410 with bad export cookie 10491162927972381973 [11418.863755] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11418.877888] LustreError: 454151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11418.988076] LustreError: 455050:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11419.046229] LustreError: 455050:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11419.176908] Lustre: server umount lustre-MDT0001 complete [11433.364788] LustreError: 455251:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [11433.378476] LustreError: 455251:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11433.607297] Lustre: server umount lustre-OST0000 complete [11444.416867] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:53:54 (1763344434) [11445.417644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11451.499816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11451.834159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11455.272766] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11456.994812] LustreError: 456080:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11457.013454] LustreError: 456080:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11462.113214] LustreError: 456080:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11462.747371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11462.976599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11466.191307] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11470.678693] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11473.957717] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11480.174697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11480.243723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11480.415815] Lustre: Found index 0 for lustre-OST0000, updating log [11480.625909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11485.632851] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11491.080561] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11497.017435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11507.170044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11507.175929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11507.201248] Lustre: Skipped 1 previous similar message [11511.425332] LustreError: 458364:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11511.428366] LustreError: 458364:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11511.588498] Lustre: server umount lustre-OST0000 complete [11513.810833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11519.457240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11519.457265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11519.469852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11519.471736] Lustre: Skipped 1 previous similar message [11519.475093] Lustre: Skipped 3 previous similar messages [11521.695658] LustreError: 458618:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11521.698718] LustreError: 458618:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11521.807702] Lustre: server umount lustre-MDT0000 complete [11524.584994] LustreError: 457151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11525.083744] LustreError: 456061:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344516 with bad export cookie 10491162927972383366 [11525.086902] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11525.090405] LustreError: 456061:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11525.407093] Lustre: server umount lustre-MDT0001 complete [11533.250325] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:55:23 (1763344523) [11534.314757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11535.324586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11536.488991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11537.431583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11538.355431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11539.162662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11539.889423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11540.712025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11541.587656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11542.541752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11543.434490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11543.461816] LNet: 1 local NIs in recovery (showing 1): 192.168.203.130@tcp [11544.456801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11545.292297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11546.140992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11546.982618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11547.792531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11548.612273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11549.359732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11550.388800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11551.208179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11552.053589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11552.841166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11553.651328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11554.432458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11555.205666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11556.041140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11556.895157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11557.801347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11558.710290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11559.549901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11560.302580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11561.145391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11561.939605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11562.832674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11563.582934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11564.322461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11565.079478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11565.811757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11566.593513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11567.323784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11568.122535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11568.913166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11569.776356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11570.613137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11571.434452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11572.282348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11573.090616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11573.839021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11574.700831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11575.454931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11576.228669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11582.566470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11582.932580] LustreError: 462212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11582.952881] LustreError: 462212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11583.011706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11586.251250] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11588.073148] LustreError: 462212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11592.722642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11595.939311] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11600.395501] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11603.637985] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11609.131361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11609.230028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11609.462870] Lustre: Found index 0 for lustre-OST0000, updating log [11611.829926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [11614.332839] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11619.323292] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11622.303302] LNet: 1 local NIs in recovery (showing 1): 192.168.203.130@tcp [11622.371178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11622.374931] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11622.417549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11622.426791] Lustre: Skipped 1 previous similar message [11626.914280] LustreError: 464470:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11626.925551] LustreError: 464470:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [11627.068867] Lustre: server umount lustre-OST0000 complete [11631.268443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11631.274246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11631.286395] Lustre: Skipped 2 previous similar messages [11631.298919] Lustre: Skipped 2 previous similar messages [11636.451323] LustreError: 462212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11636.560388] Lustre: server umount lustre-MDT0000 complete [11640.705354] LustreError: 462192:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344632 with bad export cookie 10491162927972384451 [11640.707792] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11640.729106] LustreError: 462192:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11641.125700] Lustre: server umount lustre-MDT0001 complete [11644.542775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11661.533947] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [11668.758853] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [11677.545607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11683.621701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11683.745388] LNet: 1 local NIs in recovery (showing 1): 192.168.203.130@tcp [11688.728317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11694.585837] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11701.290418] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:58:11 (1763344691) [11716.943596] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [11719.605726] Key type lgssc unregistered [11719.986469] LNet: 468441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11720.004063] LNetError: 468441:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11752.806826] LNet: Removed LNI 192.168.203.130@tcp [11753.510482] Key type .llcrypt unregistered [11753.512072] Key type ._llcrypt unregistered [11769.001196] Key type ._llcrypt registered [11769.002987] Key type .llcrypt registered [11769.099844] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [11770.087692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11770.108489] alg: No test for adler32 (adler32-zlib) [11771.214062] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [11771.481781] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [11773.199708] Key type lgssc registered [11774.269727] Lustre: Echo OBD driver; http://www.lustre.org/ [11778.062041] LNet: Added LNI 192.168.203.130@tcp100 [8/256/0/180] [11778.827631] LNet: Added LNI 192.168.203.130@tcp101 [8/256/0/180] [11779.531343] LNet: Added LNI 192.168.203.130@tcp102 [8/256/0/180] [11780.124812] LNet: Added LNI 192.168.203.130@tcp103 [8/256/0/180] [11780.738105] LNet: Added LNI 192.168.203.130@tcp104 [8/256/0/180] [11781.404957] LNet: Added LNI 192.168.203.130@tcp105 [8/256/0/180] [11782.040640] LNet: Added LNI 192.168.203.130@tcp106 [8/256/0/180] [11782.711414] LNet: Added LNI 192.168.203.130@tcp107 [8/256/0/180] [11783.309271] LNet: Added LNI 192.168.203.130@tcp108 [8/256/0/180] [11783.947100] LNet: Added LNI 192.168.203.130@tcp109 [8/256/0/180] [11784.573710] LNet: Added LNI 192.168.203.130@tcp110 [8/256/0/180] [11785.128963] LNet: Added LNI 192.168.203.130@tcp111 [8/256/0/180] [11785.710561] LNet: Added LNI 192.168.203.130@tcp112 [8/256/0/180] [11786.344164] LNet: Added LNI 192.168.203.130@tcp113 [8/256/0/180] [11786.938578] LNet: Added LNI 192.168.203.130@tcp114 [8/256/0/180] [11787.621292] LNet: Added LNI 192.168.203.130@tcp115 [8/256/0/180] [11788.207028] LNet: Added LNI 192.168.203.130@tcp116 [8/256/0/180] [11788.856433] LNet: Added LNI 192.168.203.130@tcp117 [8/256/0/180] [11789.641188] LNet: Added LNI 192.168.203.130@tcp118 [8/256/0/180] [11790.388186] LNet: Added LNI 192.168.203.130@tcp119 [8/256/0/180] [11790.969233] LNet: Added LNI 192.168.203.130@tcp120 [8/256/0/180] [11791.632197] LNet: Added LNI 192.168.203.130@tcp121 [8/256/0/180] [11792.247058] LNet: Added LNI 192.168.203.130@tcp122 [8/256/0/180] [11792.832915] LNet: Added LNI 192.168.203.130@tcp123 [8/256/0/180] [11793.484526] LNet: Added LNI 192.168.203.130@tcp124 [8/256/0/180] [11794.182993] LNet: Added LNI 192.168.203.130@tcp125 [8/256/0/180] [11794.928848] LNet: Added LNI 192.168.203.130@tcp126 [8/256/0/180] [11795.607865] LNet: Added LNI 192.168.203.130@tcp127 [8/256/0/180] [11796.238800] LNet: Added LNI 192.168.203.130@tcp128 [8/256/0/180] [11796.881640] LNet: Added LNI 192.168.203.130@tcp129 [8/256/0/180] [11797.578081] LNet: Added LNI 192.168.203.130@tcp130 [8/256/0/180] [11798.204188] LNet: Added LNI 192.168.203.130@tcp131 [8/256/0/180] [11798.848732] LNet: Added LNI 192.168.203.130@tcp132 [8/256/0/180] [11799.547715] LNet: Added LNI 192.168.203.130@tcp133 [8/256/0/180] [11800.140780] LNet: Added LNI 192.168.203.130@tcp134 [8/256/0/180] [11800.815775] LNet: Added LNI 192.168.203.130@tcp135 [8/256/0/180] [11806.523534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11806.551958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11806.562480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11807.641358] Lustre: 471449:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.130@tcp: can't add NID 192.168.203.130@tcp130: rc = -75 [11807.711941] LNetError: 468909:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11807.718735] LNetError: Protocol error connecting to 192.168.203.130@tcp118 at host 192.168.203.130:988: is it running a compatible version of Lustre? [11807.722992] LNetError: Skipped 3 previous similar messages [11807.805552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11807.838261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11807.891635] Lustre: lustre-MDT0000: new disk, initializing [11807.996606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11808.016385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11808.351599] LNet: 9 local NIs in recovery (showing 9): 192.168.203.130@tcp122, 192.168.203.130@tcp123, 192.168.203.130@tcp125, 192.168.203.130@tcp124, 192.168.203.130@tcp121, 192.168.203.130@tcp129, 192.168.203.130@tcp128, 192.168.203.130@tcp126, 192.168.203.130@tcp127 [11808.353502] LNetError: 471481:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11808.390224] LNetError: 471481:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 13 previous similar messages [11808.394240] LNetError: Protocol error connecting to 192.168.203.130@tcp at host 192.168.203.130:988: is it running a compatible version of Lustre? [11808.399256] LNetError: Skipped 10 previous similar messages [11811.335699] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11820.155546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11820.219843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11820.293975] Lustre: 471466:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11820.360381] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11820.362944] Lustre: Skipped 1 previous similar message [11820.431124] Lustre: lustre-MDT0001: new disk, initializing [11820.458725] Lustre: 472256:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.130@tcp: can't add NID 192.168.203.130@tcp130: rc = -75 [11820.463192] Lustre: 472256:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11820.605788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11820.656166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11820.671317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11823.644965] Lustre: 472463:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.130@tcp: can't add NID 192.168.203.130@tcp130: rc = -75 [11823.654737] Lustre: 472463:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11823.873741] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11829.224661] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11832.047146] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11837.089206] LNetError: 468911:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11837.099554] LNetError: 468911:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [11837.117851] LNetError: Protocol error connecting to 192.168.203.130@tcp at host 192.168.203.130:988: is it running a compatible version of Lustre? [11837.875466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11837.958103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11838.195499] Lustre: lustre-OST0000: new disk, initializing [11838.199911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11838.297451] Lustre: 473226:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.130@tcp: can't add NID 192.168.203.130@tcp130: rc = -75 [11838.302439] Lustre: 473226:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [11838.351627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11839.137326] LNetError: 471486:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11839.150357] LNetError: Protocol error connecting to 192.168.203.130@tcp118 at host 192.168.203.130:988: is it running a compatible version of Lustre? [11843.233918] LNetError: 471472:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11843.255764] LNetError: 471472:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [11843.271502] LNetError: Protocol error connecting to 192.168.203.130@tcp124 at host 192.168.203.130:988: is it running a compatible version of Lustre? [11843.272461] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11843.284198] LNetError: Skipped 1 previous similar message [11845.677857] Lustre: 473662:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.130@tcp: can't add NID 192.168.203.130@tcp130: rc = -75 [11845.697844] Lustre: 473662:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11845.759847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11845.765762] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11845.834764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11849.951448] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11852.448559] LNetError: 468910:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11852.474055] LNetError: 468910:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 3 previous similar messages [11852.492366] LNetError: Protocol error connecting to 192.168.203.130@tcp118 at host 192.168.203.130:988: is it running a compatible version of Lustre? [11852.505067] LNetError: Skipped 3 previous similar messages [11861.478303] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11861.478873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11861.486771] LustreError: Skipped 1 previous similar message [11861.490883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11861.513037] Lustre: Skipped 1 previous similar message [11863.558979] LustreError: 474059:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11863.708780] Lustre: server umount lustre-OST0000 complete [11869.856404] LNetError: 471483:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11869.881331] LNetError: 471483:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 6 previous similar messages [11869.894839] LNetError: Protocol error connecting to 192.168.203.130@tcp119 at host 192.168.203.130:988: is it running a compatible version of Lustre? [11869.903730] LNetError: Skipped 6 previous similar messages [11871.713158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11871.716307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11871.724782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11871.739142] Lustre: Skipped 1 previous similar message [11871.747615] Lustre: Skipped 1 previous similar message [11872.920659] LustreError: 474260:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11872.932838] LustreError: 474260:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11873.116318] Lustre: server umount lustre-MDT0000 complete [11876.574709] LustreError: 471459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344868 with bad export cookie 10594994680917448697 [11876.581693] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11876.609533] LustreError: 471459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11876.718872] LustreError: 474475:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11876.722443] LustreError: 474475:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11876.890069] Lustre: server umount lustre-MDT0001 complete [11885.966722] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [11888.382953] Key type lgssc unregistered [11888.807537] LNet: 475015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11888.823264] LNetError: 475015:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11888.832768] LNet: Removed LNI 192.168.203.130@tcp [11889.887165] LNet: Removed LNI 192.168.203.130@tcp100 [11889.895836] LNetError: 475015:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11889.908241] LNetError: 475015:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11889.925968] LNet: Removed LNI 192.168.203.130@tcp101 [11890.975183] LNet: Removed LNI 192.168.203.130@tcp102 [11890.977499] LNetError: 475015:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11890.992162] LNetError: 475015:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11891.001622] LNet: Removed LNI 192.168.203.130@tcp103 [11892.063137] LNet: Removed LNI 192.168.203.130@tcp104 [11893.087108] LNet: Removed LNI 192.168.203.130@tcp105 [11893.090925] LNetError: 475015:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11893.104367] LNetError: 475015:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11893.114414] LNet: Removed LNI 192.168.203.130@tcp106 [11894.177436] LNet: Removed LNI 192.168.203.130@tcp107 [11895.199156] LNet: Removed LNI 192.168.203.130@tcp108 [11896.223165] LNet: Removed LNI 192.168.203.130@tcp109 [11897.247175] LNet: Removed LNI 192.168.203.130@tcp110 [11897.255978] LNetError: 475015:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11897.264399] LNetError: 475015:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11897.275133] LNet: Removed LNI 192.168.203.130@tcp111 [11898.335817] LNet: Removed LNI 192.168.203.130@tcp112 [11899.359207] LNet: Removed LNI 192.168.203.130@tcp113 [11900.383218] LNet: Removed LNI 192.168.203.130@tcp114 [11901.407296] LNet: Removed LNI 192.168.203.130@tcp115 [11902.431146] LNet: Removed LNI 192.168.203.130@tcp116 [11903.455234] LNet: Removed LNI 192.168.203.130@tcp117 [11904.479192] LNet: Removed LNI 192.168.203.130@tcp118 [11905.503507] LNet: Removed LNI 192.168.203.130@tcp119 [11905.507714] LNetError: 475015:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11905.517661] LNetError: 475015:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [11905.532200] LNet: Removed LNI 192.168.203.130@tcp120 [11906.591233] LNet: Removed LNI 192.168.203.130@tcp121 [11907.615206] LNet: Removed LNI 192.168.203.130@tcp122 [11908.639227] LNet: Removed LNI 192.168.203.130@tcp123 [11909.663210] LNet: Removed LNI 192.168.203.130@tcp124 [11910.687235] LNet: Removed LNI 192.168.203.130@tcp125 [11911.711344] LNet: Removed LNI 192.168.203.130@tcp126 [11912.736861] LNet: Removed LNI 192.168.203.130@tcp127 [11913.759199] LNet: Removed LNI 192.168.203.130@tcp128 [11914.783205] LNet: Removed LNI 192.168.203.130@tcp129 [11914.785327] LNet: Removed LNI 192.168.203.130@tcp130 [11914.790216] LNet: Removed LNI 192.168.203.130@tcp131 [11914.799901] LNet: Removed LNI 192.168.203.130@tcp132 [11914.804649] LNet: Removed LNI 192.168.203.130@tcp133 [11914.806775] LNet: Removed LNI 192.168.203.130@tcp134 [11914.821650] LNet: Removed LNI 192.168.203.130@tcp135 [11915.597428] Key type .llcrypt unregistered [11915.605313] Key type ._llcrypt unregistered [11934.224286] Key type ._llcrypt registered [11934.226054] Key type .llcrypt registered [11934.302590] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [11947.795781] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [11948.628778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11948.798778] alg: No test for adler32 (adler32-zlib) [11949.827717] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [11950.010935] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [11951.688188] Key type lgssc registered [11952.658435] Lustre: Echo OBD driver; http://www.lustre.org/ [11959.998645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11965.777547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11970.946697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11976.681609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11982.795259] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 21:02:53 (1763344973) [12000.373401] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [12003.289290] Key type lgssc unregistered [12003.633128] LNet: 479067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12003.652915] LNetError: 479067:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12003.666288] LNet: Removed LNI 192.168.203.130@tcp [12004.505439] Key type .llcrypt unregistered [12004.511564] Key type ._llcrypt unregistered [12020.077063] Key type ._llcrypt registered [12020.079206] Key type .llcrypt registered [12020.155887] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [12020.824601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12020.842079] alg: No test for adler32 (adler32-zlib) [12021.899397] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [12022.160577] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [12023.914496] Key type lgssc registered [12024.978358] Lustre: Echo OBD driver; http://www.lustre.org/ [12034.311208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12034.409838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12034.434637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12035.716676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12035.744460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12035.796172] Lustre: lustre-MDT0000: new disk, initializing [12035.899221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12035.920649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12040.206817] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12050.919228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12051.022570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12051.140825] Lustre: 480262:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12051.191624] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12051.200328] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12051.244691] Lustre: Skipped 1 previous similar message [12051.366569] Lustre: lustre-MDT0001: new disk, initializing [12051.492247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12051.546377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12051.565635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12055.004476] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12061.949168] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12065.710361] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12071.530520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12071.573986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12071.820445] Lustre: lustre-OST0000: new disk, initializing [12071.826759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12071.935359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12076.182351] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12077.633394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12077.638568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12077.732701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12083.191479] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12086.172792] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing lnet_if_list [12089.924693] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [12089.943430] LNet: Added LNI 192.168.203.130@tcp10 [8/256/0/180] [12094.307888] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [12094.360419] LNet: Added LNI 192.168.203.130@tcp11 [8/256/0/180] [12098.277736] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [12098.312590] LNet: Added LNI 192.168.203.130@tcp12 [8/256/0/180] [12102.343956] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12102.391471] LNet: Added LNI 192.168.203.130@tcp13 [8/256/0/180] [12106.596709] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12106.627102] LNet: Added LNI 192.168.203.130@tcp14 [8/256/0/180] [12110.802120] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12110.836290] LNet: Added LNI 192.168.203.130@tcp15 [8/256/0/180] [12115.390814] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12115.416441] LNet: Added LNI 192.168.203.130@tcp16 [8/256/0/180] [12119.820910] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12119.856823] LNet: Added LNI 192.168.203.130@tcp17 [8/256/0/180] [12124.409895] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12124.454537] LNet: Added LNI 192.168.203.130@tcp18 [8/256/0/180] [12128.833965] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12128.855746] LNet: Added LNI 192.168.203.130@tcp19 [8/256/0/180] [12133.515310] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12133.545847] LNet: Added LNI 192.168.203.130@tcp20 [8/256/0/180] [12137.971652] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12138.003935] LNet: Added LNI 192.168.203.130@tcp21 [8/256/0/180] [12142.487262] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12142.524437] LNet: Added LNI 192.168.203.130@tcp22 [8/256/0/180] [12147.099758] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12147.124992] LNet: Added LNI 192.168.203.130@tcp23 [8/256/0/180] [12151.675531] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12151.710060] LNet: Added LNI 192.168.203.130@tcp24 [8/256/0/180] [12156.629546] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12156.650933] LNet: Added LNI 192.168.203.130@tcp25 [8/256/0/180] [12161.483116] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12161.501743] LNet: Added LNI 192.168.203.130@tcp26 [8/256/0/180] [12166.056690] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12166.081243] LNet: Added LNI 192.168.203.130@tcp27 [8/256/0/180] [12166.097247] LNetError: 479522:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12166.114379] LNetError: Protocol error connecting to 192.168.203.130@tcp26 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12170.860759] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12170.886649] LNet: Added LNI 192.168.203.130@tcp28 [8/256/0/180] [12170.899909] LNetError: 479519:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12170.903617] LNetError: Protocol error connecting to 192.168.203.130@tcp26 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12170.932924] LNetError: Skipped 1 previous similar message [12171.877604] LNet: 1 local NIs in recovery (showing 1): 192.168.203.130@tcp27 [12172.277160] LNetError: 479522:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12172.292413] LNetError: 479522:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 2 previous similar messages [12172.310978] LNetError: Protocol error connecting to 192.168.203.130@tcp28 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12172.317050] LNetError: Skipped 1 previous similar message [12175.758864] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12175.796978] LNet: Added LNI 192.168.203.130@tcp29 [8/256/0/180] [12175.982705] LNetError: 479521:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12175.993323] LNetError: 479521:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 2 previous similar messages [12176.000688] LNetError: Protocol error connecting to 192.168.203.130@tcp28 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12176.021181] LNetError: Skipped 2 previous similar messages [12180.408753] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [12180.430920] LNet: Added LNI 192.168.203.130@tcp30 [8/256/0/180] [12180.439633] LNetError: 479520:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12180.457669] LNetError: 479520:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [12180.466578] LNetError: Protocol error connecting to 192.168.203.130@tcp29 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12180.474946] LNetError: Skipped 1 previous similar message [12185.646494] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [12185.678280] LNet: Added LNI 192.168.203.130@tcp31 [8/256/0/180] [12190.740653] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [12190.771373] LNet: Added LNI 192.168.203.130@tcp32 [8/256/0/180] [12190.780961] LNetError: 479521:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12190.796985] LNetError: 479521:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 5 previous similar messages [12190.808947] LNetError: Protocol error connecting to 192.168.203.130@tcp30 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12190.816661] LNetError: Skipped 5 previous similar messages [12195.411986] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [12195.434350] LNet: Added LNI 192.168.203.130@tcp33 [8/256/0/180] [12200.869604] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12200.893086] LNet: Added LNI 192.168.203.130@tcp34 [8/256/0/180] [12206.942798] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12206.976380] LNet: Added LNI 192.168.203.130@tcp35 [8/256/0/180] [12207.713557] LNetError: 479521:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12207.725511] LNetError: 479521:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 21 previous similar messages [12207.735587] LNetError: Protocol error connecting to 192.168.203.130@tcp32 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12207.754445] LNetError: Skipped 21 previous similar messages [12213.311527] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12213.341930] LNet: Added LNI 192.168.203.130@tcp36 [8/256/0/180] [12219.492379] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12219.532092] LNet: Added LNI 192.168.203.130@tcp37 [8/256/0/180] [12226.310605] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12226.335767] LNet: Added LNI 192.168.203.130@tcp38 [8/256/0/180] [12232.287224] LNet: 12 local NIs in recovery (showing 12): 192.168.203.130@tcp32, 192.168.203.130@tcp38, 192.168.203.130@tcp37, 192.168.203.130@tcp35, 192.168.203.130@tcp36, 192.168.203.130@tcp31, 192.168.203.130@tcp30, 192.168.203.130@tcp33, 192.168.203.130@tcp34, 192.168.203.130@tcp29, 192.168.203.130@tcp28, 192.168.203.130@tcp27 [12232.844907] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12232.872503] LNet: Added LNI 192.168.203.130@tcp39 [8/256/0/180] [12238.836946] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12238.863110] LNet: Added LNI 192.168.203.130@tcp40 [8/256/0/180] [12240.364996] LNetError: 479521:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12240.390298] LNetError: 479521:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 36 previous similar messages [12240.409091] LNetError: Protocol error connecting to 192.168.203.130@tcp40 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12240.417132] LNetError: Skipped 36 previous similar messages [12245.372917] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12245.394847] LNet: Added LNI 192.168.203.130@tcp41 [8/256/0/180] [12251.162807] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12251.187757] LNet: Added LNI 192.168.203.130@tcp42 [8/256/0/180] [12256.661532] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12256.692859] LNet: Added LNI 192.168.203.130@tcp43 [8/256/0/180] [12263.433440] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12263.459916] LNet: Added LNI 192.168.203.130@tcp44 [8/256/0/180] [12269.505158] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12269.532950] LNet: Added LNI 192.168.203.130@tcp45 [8/256/0/180] [12275.623781] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12275.647023] LNet: Added LNI 192.168.203.130@tcp46 [8/256/0/180] [12281.808760] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12281.848791] LNet: Added LNI 192.168.203.130@tcp47 [8/256/0/180] [12288.471423] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12288.508966] LNet: Added LNI 192.168.203.130@tcp48 [8/256/0/180] [12295.167022] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12295.193824] LNet: Added LNI 192.168.203.130@tcp49 [8/256/0/180] [12302.327908] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12302.345209] LNet: Added LNI 192.168.203.130@tcp50 [8/256/0/180] [12304.992895] LNetError: 479521:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12305.004876] LNetError: 479521:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 68 previous similar messages [12305.020658] LNetError: Protocol error connecting to 192.168.203.130@tcp26 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12305.037601] LNetError: Skipped 68 previous similar messages [12323.812028] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12323.816870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12323.821961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12323.854838] Lustre: Skipped 1 previous similar message [12323.858500] LustreError: Skipped 1 previous similar message [12326.339049] LustreError: 504530:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12326.546530] Lustre: server umount lustre-OST0000 complete [12333.025909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12333.033492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12333.047930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12333.050589] Lustre: Skipped 1 previous similar message [12334.055914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12336.269237] LustreError: 504732:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12336.274023] LustreError: 504732:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12336.334529] Lustre: server umount lustre-MDT0000 complete [12338.872549] LustreError: 480251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345330 with bad export cookie 7759460085791183547 [12338.872892] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12338.883178] LustreError: 480251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12338.959067] LustreError: 504933:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12338.966385] LustreError: 504933:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12339.160507] Lustre: server umount lustre-MDT0001 complete [12346.963741] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [12348.738667] Key type lgssc unregistered [12349.014479] LNet: 505472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12349.020838] LNetError: 505472:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12349.027980] LNet: Removed LNI 192.168.203.130@tcp [12350.047183] LNet: Removed LNI 192.168.203.130@tcp10 [12350.050964] LNetError: 505472:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12350.056774] LNetError: 505472:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12350.066431] LNet: Removed LNI 192.168.203.130@tcp11 [12351.071169] LNet: Removed LNI 192.168.203.130@tcp12 [12351.073415] LNetError: 505472:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12351.081489] LNetError: 505472:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12351.089381] LNet: Removed LNI 192.168.203.130@tcp13 [12352.095717] LNet: Removed LNI 192.168.203.130@tcp14 [12353.119214] LNet: Removed LNI 192.168.203.130@tcp15 [12353.122228] LNetError: 505472:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12353.125514] LNetError: 505472:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12353.131346] LNet: Removed LNI 192.168.203.130@tcp16 [12354.143395] LNet: Removed LNI 192.168.203.130@tcp17 [12355.167184] LNet: Removed LNI 192.168.203.130@tcp18 [12356.191175] LNet: Removed LNI 192.168.203.130@tcp19 [12357.216365] LNet: Removed LNI 192.168.203.130@tcp20 [12357.223384] LNetError: 505472:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12357.231497] LNetError: 505472:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12357.236728] LNet: Removed LNI 192.168.203.130@tcp21 [12358.303145] LNet: Removed LNI 192.168.203.130@tcp22 [12359.329664] LNet: Removed LNI 192.168.203.130@tcp23 [12360.351369] LNet: Removed LNI 192.168.203.130@tcp24 [12361.375192] LNet: Removed LNI 192.168.203.130@tcp25 [12362.399176] LNet: Removed LNI 192.168.203.130@tcp26 [12362.401276] LNet: Removed LNI 192.168.203.130@tcp27 [12362.404627] LNet: Removed LNI 192.168.203.130@tcp28 [12362.407339] LNet: Removed LNI 192.168.203.130@tcp29 [12362.409270] LNet: Removed LNI 192.168.203.130@tcp30 [12362.412658] LNet: Removed LNI 192.168.203.130@tcp31 [12362.414808] LNet: Removed LNI 192.168.203.130@tcp32 [12362.417386] LNet: Removed LNI 192.168.203.130@tcp33 [12362.419454] LNet: Removed LNI 192.168.203.130@tcp34 [12362.421355] LNet: Removed LNI 192.168.203.130@tcp35 [12362.423142] LNet: Removed LNI 192.168.203.130@tcp36 [12362.424932] LNet: Removed LNI 192.168.203.130@tcp37 [12362.426685] LNet: Removed LNI 192.168.203.130@tcp38 [12362.429816] LNet: Removed LNI 192.168.203.130@tcp39 [12362.432658] LNet: Removed LNI 192.168.203.130@tcp40 [12362.435102] LNet: Removed LNI 192.168.203.130@tcp41 [12363.487171] LNet: Removed LNI 192.168.203.130@tcp42 [12363.489325] LNet: Removed LNI 192.168.203.130@tcp43 [12363.494234] LNet: Removed LNI 192.168.203.130@tcp44 [12363.496522] LNet: Removed LNI 192.168.203.130@tcp45 [12364.511224] LNet: Removed LNI 192.168.203.130@tcp46 [12364.515532] LNet: Removed LNI 192.168.203.130@tcp47 [12364.522476] LNet: Removed LNI 192.168.203.130@tcp48 [12364.524860] LNet: Removed LNI 192.168.203.130@tcp49 [12364.537447] LNet: Removed LNI 192.168.203.130@tcp50 [12365.176577] Key type .llcrypt unregistered [12365.178412] Key type ._llcrypt unregistered [12379.233924] Key type ._llcrypt registered [12379.235392] Key type .llcrypt registered [12379.303289] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [12380.002187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12380.016299] alg: No test for adler32 (adler32-zlib) [12380.977784] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [12381.129321] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [12382.759293] Key type lgssc registered [12383.676793] Lustre: Echo OBD driver; http://www.lustre.org/ [12390.388992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12390.402926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12391.744815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12394.731238] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12401.429449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12401.729987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12404.778845] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12410.148320] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12413.175862] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12418.582797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12418.905264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12423.738396] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12428.479125] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12446.899822] loop0: detected capacity change from 0 to 204800000 [12447.112896] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12447.123212] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12447.131110] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12447.183057] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12451.656816] loop0: detected capacity change from 0 to 204800000 [12451.783041] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12451.802131] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12451.812628] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12451.899892] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12455.302828] loop0: detected capacity change from 0 to 204800000 [12459.415766] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12459.473779] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12459.574836] Lustre: fs15246-OST0000: new disk, initializing [12459.580798] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12459.624240] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12463.679670] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12470.890280] loop1: detected capacity change from 0 to 204800000 [12475.261248] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12475.310832] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12475.374617] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12475.429494] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12475.509746] Lustre: fs15246-MDT0000: new disk, initializing [12475.643671] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12475.685933] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12475.797505] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12476.832328] Lustre: 511097:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000ed470672 x1849001636833536/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763345484 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12478.755762] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12485.995929] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12486.003953] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12486.021817] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12497.381473] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12497.397279] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12501.009046] LustreError: 512220:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [12501.057752] Lustre: server umount fs15246-MDT0000 complete [12502.501242] LustreError: 508858:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12502.540537] LustreError: 508858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12507.319947] LustreError: 512720:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [12507.330951] LustreError: 512720:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12507.361552] Lustre: server umount fs15246-OST0000 complete [12515.296047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12515.299717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12515.331452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12517.861237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12517.869343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12522.986561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12522.993747] Lustre: Skipped 2 previous similar messages [12528.101871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12528.106261] Lustre: Skipped 1 previous similar message [12529.119853] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12529.169518] LustreError: 513224:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12529.175149] LustreError: 513224:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [12529.282697] Lustre: server umount lustre-OST0000 complete [12533.216321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12533.226671] Lustre: Skipped 1 previous similar message [12538.147940] LustreError: 513428:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12538.155540] LustreError: 513428:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12538.247158] Lustre: server umount lustre-MDT0000 complete [12538.348294] LustreError: 506696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12538.377775] LustreError: 506696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12541.646101] LustreError: 513225:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345533 with bad export cookie 4440671865454375139 [12541.651111] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12542.036677] Lustre: server umount lustre-MDT0001 complete [12550.254347] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [12552.636909] Key type lgssc unregistered [12552.962831] LNet: 514161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12552.978412] LNetError: 514161:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12553.024289] LNet: Removed LNI 192.168.203.130@tcp [12554.373096] Key type .llcrypt unregistered [12554.377566] Key type ._llcrypt unregistered [12572.873954] Key type ._llcrypt registered [12572.875809] Key type .llcrypt registered [12572.986597] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [12573.903324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12573.932598] alg: No test for adler32 (adler32-zlib) [12574.904826] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [12575.133465] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [12576.783321] Key type lgssc registered [12577.942915] Lustre: Echo OBD driver; http://www.lustre.org/ [12585.135905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12588.998733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12593.046628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12597.545724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12610.619131] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [12617.091144] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [12625.180733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12630.911752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12635.849368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12641.477979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12646.633158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12646.665461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12646.676381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12647.815961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12647.850240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12647.923351] Lustre: lustre-MDT0000: new disk, initializing [12647.998435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12648.018145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12650.749717] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12658.464615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12658.529175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12658.591259] Lustre: 518594:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12658.636094] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12658.639589] Lustre: Skipped 1 previous similar message [12658.711575] Lustre: lustre-MDT0001: new disk, initializing [12658.813898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12658.845183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12658.856115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12662.078278] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12667.807581] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12671.214866] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12677.000674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12677.064727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12677.235272] Lustre: lustre-OST0000: new disk, initializing [12677.244205] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12677.326249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12681.294520] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12687.263625] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12687.403623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12687.410437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12687.520824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12690.675441] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12690.829233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12693.396426] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12693.564663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12698.090507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12698.094435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12698.097155] LustreError: Skipped 1 previous similar message [12698.098782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12698.102835] Lustre: Skipped 1 previous similar message [12698.118240] Lustre: Skipped 1 previous similar message [12703.204100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12708.325508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12708.333360] Lustre: Skipped 2 previous similar messages [12709.343237] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12709.381887] LustreError: 521528:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12709.557595] Lustre: server umount lustre-OST0000 complete [12713.445855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12713.448460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12713.451406] Lustre: Skipped 1 previous similar message [12713.471211] Lustre: Skipped 1 previous similar message [12718.564805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12718.568513] Lustre: Skipped 1 previous similar message [12718.652996] LustreError: 521733:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12718.663459] LustreError: 521733:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12718.751720] Lustre: server umount lustre-MDT0000 complete [12721.897930] LustreError: 518585:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345713 with bad export cookie 2856379310961134886 [12721.906444] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12721.915952] LustreError: 518585:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12722.084831] LustreError: 521934:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12722.094194] LustreError: 521934:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12722.291988] Lustre: server umount lustre-MDT0001 complete [12742.571291] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [12749.446561] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [12757.145083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12762.658871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12768.494842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12774.389727] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12775.877825] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12523 sec ========= 21:16:05 (1763345765) [12777.323687] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:16:07 (1763345767) === [12780.159381] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:16:10 (1763345770) === [12814.839978] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [12817.135728] Key type lgssc unregistered [12817.439723] LNet: 526795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12817.458113] LNetError: 526795:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12818.475323] LNet: Removed LNI 192.168.203.130@tcp [12819.269355] Key type .llcrypt unregistered [12819.277271] Key type ._llcrypt unregistered