[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488118218 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2496MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003128] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008297] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011141] LSM: Security Framework initializing [ 0.012051] Yama: becoming mindful. [ 0.013030] SELinux: Initializing. [ 0.014075] *** VALIDATE selinux *** [ 0.022791] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027783] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030111] *** VALIDATE tmpfs *** [ 0.031429] *** VALIDATE proc *** [ 0.033249] *** VALIDATE cgroup *** [ 0.034022] *** VALIDATE cgroup2 *** [ 0.035265] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039010] Spectre V2 : User space: Vulnerable [ 0.040007] Speculative Store Bypass: Vulnerable [ 0.042893] debug: unmapping init [mem 0xffffffffa2c59000-0xffffffffa2c60fff] [ 0.044814] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045866] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047016] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049016] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051010] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053295] rcu: Hierarchical SRCU implementation. [ 0.055516] smp: Bringing up secondary CPUs ... [ 0.056585] x86: Booting SMP configuration: [ 0.057023] .... node #0, CPUs: #1 #2 #3 [ 0.060083] smp: Brought up 1 node, 4 CPUs [ 0.062021] smpboot: Max logical packages: 1 [ 0.063017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.161229] node 0 deferred pages initialised in 92ms [ 0.164274] devtmpfs: initialized [ 0.165271] x86/mm: Memory block size: 128MB [ 0.169948] gcov: version magic: 0x41383552 [ 0.171396] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.174066] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.175258] pinctrl core: initialized pinctrl subsystem [ 0.176178] [ 0.176765] ************************************************************* [ 0.177014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178010] ** ** [ 0.179013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.180014] ** ** [ 0.181013] ** This means that this kernel is built to expose internal ** [ 0.182016] ** IOMMU data structures, which may compromise security on ** [ 0.183014] ** your system. ** [ 0.184014] ** ** [ 0.185013] ** If you see this message and you are not debugging the ** [ 0.186014] ** kernel, report this immediately to your vendor! ** [ 0.187013] ** ** [ 0.188013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189015] ************************************************************* [ 0.190709] NET: Registered protocol family 16 [ 0.191427] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.194440] cpuidle: using governor menu [ 0.197347] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.200386] PCI: Using configuration type 1 for base access [ 0.202121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.211127] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.213034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.216200] cryptd: max_cpu_qlen set to 1000 [ 0.219225] ACPI: Added _OSI(Module Device) [ 0.221019] ACPI: Added _OSI(Processor Device) [ 0.222011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.223014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.227600] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.234182] ACPI: Interpreter enabled [ 0.236054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.237010] ACPI: Using IOAPIC for interrupt routing [ 0.239755] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243391] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.252989] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.255034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.257015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.259070] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.264196] acpiphp: Slot [2] registered [ 0.265106] acpiphp: Slot [3] registered [ 0.267098] acpiphp: Slot [4] registered [ 0.268116] acpiphp: Slot [5] registered [ 0.270121] acpiphp: Slot [6] registered [ 0.271077] acpiphp: Slot [7] registered [ 0.272078] acpiphp: Slot [8] registered [ 0.273094] acpiphp: Slot [9] registered [ 0.275105] acpiphp: Slot [10] registered [ 0.276107] acpiphp: Slot [11] registered [ 0.277104] acpiphp: Slot [12] registered [ 0.279102] acpiphp: Slot [13] registered [ 0.280092] acpiphp: Slot [14] registered [ 0.282168] acpiphp: Slot [15] registered [ 0.284132] acpiphp: Slot [16] registered [ 0.285104] acpiphp: Slot [17] registered [ 0.286100] acpiphp: Slot [18] registered [ 0.288239] acpiphp: Slot [19] registered [ 0.289127] acpiphp: Slot [20] registered [ 0.291126] acpiphp: Slot [21] registered [ 0.292106] acpiphp: Slot [22] registered [ 0.294125] acpiphp: Slot [23] registered [ 0.295107] acpiphp: Slot [24] registered [ 0.297145] acpiphp: Slot [25] registered [ 0.299093] acpiphp: Slot [26] registered [ 0.300102] acpiphp: Slot [27] registered [ 0.302166] acpiphp: Slot [28] registered [ 0.304133] acpiphp: Slot [29] registered [ 0.305111] acpiphp: Slot [30] registered [ 0.306113] acpiphp: Slot [31] registered [ 0.308105] PCI host bridge to bus 0000:00 [ 0.309026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.312055] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.314030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.316038] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.318043] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.321035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.323222] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.328866] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.331648] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.340816] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.346027] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.349022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.351023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.354020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.357473] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.359687] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.362053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.365725] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.371023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.385024] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.390867] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.395811] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.403015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.410019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.432024] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.444677] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.453022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.462022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.485025] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.496254] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.503019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.509016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.533018] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.542223] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.551017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.557015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.576021] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.590644] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.605022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.612018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.627017] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.637755] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.647024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.653025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.671017] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.683919] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.686416] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.689414] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.691388] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.694257] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.698028] iommu: Default domain type: Passthrough [ 0.700415] SCSI subsystem initialized [ 0.701128] ACPI: bus type USB registered [ 0.703100] usbcore: registered new interface driver usbfs [ 0.705112] usbcore: registered new interface driver hub [ 0.708088] usbcore: registered new device driver usb [ 0.710392] pps_core: LinuxPPS API ver. 1 registered [ 0.711019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.715058] PTP clock support registered [ 0.717080] EDAC MC: Ver: 3.0.0 [ 0.718446] PCI: Using ACPI for IRQ routing [ 0.720807] NetLabel: Initializing [ 0.722009] NetLabel: domain hash size = 128 [ 0.723009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.725115] NetLabel: unlabeled traffic allowed by default [ 0.727145] vgaarb: loaded [ 0.729260] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.730015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.736389] clocksource: Switched to clocksource kvm-clock [ 0.846405] VFS: Disk quotas dquot_6.6.0 [ 0.848116] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.850622] *** VALIDATE ramfs *** [ 0.852068] *** VALIDATE hugetlbfs *** [ 0.853641] pnp: PnP ACPI init [ 0.855751] pnp: PnP ACPI: found 6 devices [ 0.873984] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.877073] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.879242] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.880881] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.883199] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.885526] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.888531] NET: Registered protocol family 2 [ 0.891154] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.895982] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.899530] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.905226] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.908644] TCP: Hash tables configured (established 65536 bind 65536) [ 0.911667] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.914451] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.916571] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.919634] NET: Registered protocol family 1 [ 0.923229] RPC: Registered named UNIX socket transport module. [ 0.925684] RPC: Registered udp transport module. [ 0.927583] RPC: Registered tcp transport module. [ 0.929561] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.931610] NET: Registered protocol family 44 [ 0.933268] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.935157] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.937361] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.939477] PCI: CLS 0 bytes, default 64 [ 0.941166] Unpacking initramfs... [ 2.385433] debug: unmapping init [mem 0xffff8abbfcc54000-0xffff8abbfffbffff] [ 2.389452] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.391649] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.394922] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.876944] Initialise system trusted keyrings [ 2.879297] Key type blacklist registered [ 2.881781] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.891428] zbud: loaded [ 2.894626] *** VALIDATE nfs *** [ 2.896066] *** VALIDATE nfs4 *** [ 2.898472] pstore: using deflate compression [ 2.901582] Platform Keyring initialized [ 3.005410] NET: Registered protocol family 38 [ 3.007625] Key type asymmetric registered [ 3.009077] Asymmetric key parser 'x509' registered [ 3.010844] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.013769] io scheduler mq-deadline registered [ 3.015356] io scheduler kyber registered [ 3.016912] io scheduler bfq registered [ 3.018588] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.021401] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.024183] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.026929] ACPI: Power Button [PWRF] [ 3.116398] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.205992] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.383207] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.478281] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.675568] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.706322] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.742519] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.747910] Non-volatile memory driver v1.3 [ 3.749735] Linux agpgart interface v0.103 [ 3.780732] virtio_blk virtio1: [vda] 133840 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.783899] vda: detected capacity change from 0 to 68526080 [ 3.798059] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.801032] vdb: detected capacity change from 0 to 1073741824 [ 3.815619] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.818482] vdc: detected capacity change from 0 to 2621440000 [ 3.833981] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.836720] vdd: detected capacity change from 0 to 2621440000 [ 3.849331] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.851409] vde: detected capacity change from 0 to 4294967296 [ 3.866907] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.870167] vdf: detected capacity change from 0 to 4294967296 [ 3.878330] libphy: Fixed MDIO Bus: probed [ 3.884604] usbcore: registered new interface driver usbserial_generic [ 3.887233] usbserial: USB Serial support registered for generic [ 3.889575] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.893857] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.895835] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.898176] mousedev: PS/2 mouse device common for all mice [ 3.901428] rtc_cmos 00:05: RTC can wake from S4 [ 3.903431] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.907558] rtc_cmos 00:05: registered as rtc0 [ 3.908935] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.909660] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.915537] intel_pstate: CPU model not supported [ 3.919357] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.920507] hid: raw HID events driver (C) Jiri Kosina [ 3.924465] usbcore: registered new interface driver usbhid [ 3.926900] usbhid: USB HID core driver [ 3.928427] drop_monitor: Initializing network drop monitor service [ 3.931158] Initializing XFRM netlink socket [ 3.933196] NET: Registered protocol family 10 [ 3.936190] Segment Routing with IPv6 [ 3.937704] NET: Registered protocol family 17 [ 3.939719] mpls_gso: MPLS GSO support [ 3.945271] RAS: Correctable Errors collector initialized. [ 3.947358] AVX version of gcm_enc/dec engaged. [ 3.948581] AES CTR mode by8 optimization enabled [ 4.039404] sched_clock: Marking stable (4039010581, 0)->(5035875613, -996865032) [ 4.043358] registered taskstats version 1 [ 4.046304] Loading compiled-in X.509 certificates [ 4.048839] zswap: loaded using pool lzo/zbud [ 4.078251] Key type big_key registered [ 4.091494] Key type encrypted registered [ 4.093608] ima: No TPM chip found, activating TPM-bypass! [ 4.095887] ima: Allocated hash algorithm: sha1 [ 4.097567] ima: No architecture policies found [ 4.099098] evm: Initialising EVM extended attributes: [ 4.100696] evm: security.selinux [ 4.101852] evm: security.ima [ 4.103091] evm: security.capability [ 4.104461] evm: HMAC attrs: 0x1 [ 4.109970] rtc_cmos 00:05: setting system clock to 2025-11-17 03:46:24 UTC (1763351184) [ 4.116303] debug: unmapping init [mem 0xffffffffa3c03000-0xffffffffa3dfffff] [ 4.119223] debug: unmapping init [mem 0xffffffffa2982000-0xffffffffa2c58fff] [ 4.126091] Write protecting the kernel read-only data: 28672k [ 4.129307] debug: unmapping init [mem 0xffffffffa1003000-0xffffffffa11fffff] [ 4.131985] debug: unmapping init [mem 0xffffffffa1914000-0xffffffffa19fffff] [ 4.170358] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.182241] systemd[1]: Detected virtualization kvm. [ 4.184695] systemd[1]: Detected architecture x86-64. [ 4.187409] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.216227] systemd[1]: No hostname configured. [ 4.218259] systemd[1]: Set hostname to . [ 4.219862] random: systemd: uninitialized urandom read (16 bytes read) [ 4.222069] systemd[1]: Initializing machine ID from random generator. [ 4.282309] random: ln: uninitialized urandom read (6 bytes read) [ 4.365258] random: systemd: uninitialized urandom read (16 bytes read) [ 4.367810] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.371803] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.376718] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.037966] device-mapper: uevent: version 1.0.3 [ 5.040603] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.851643] random: fast init done [ 5.867851] virtio_net virtio0 ens2: renamed from eth0 [ 5.916669] scsi host0: ata_piix [ 5.955118] scsi host1: ata_piix [ 5.956846] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.959382] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.687762] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.618369] random: crng init done [ 10.619443] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.206344] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.408162] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.678813] SELinux: Disabled at runtime. [ 12.741532] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.749363] systemd[1]: Detected virtualization kvm. [ 12.751262] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.274376] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.277791] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.282706] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.288360] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.292303] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.299881] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.304250] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ 13.420782] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.806866] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.142884] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.150858] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.287884] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.299181] EDAC sbridge: Ver: 1.1.2 [ 16.272516] Key type dns_resolver registered [ 16.613759] NFS: Registering the id_resolver key type [ 16.615625] Key type id_resolver registered [ 16.617703] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg229-server login: [ 42.895599] libcfs: loading out-of-tree module taints kernel. [ 42.911519] Key type ._llcrypt registered [ 42.913231] Key type .llcrypt registered [ 42.969455] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 51.498801] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 52.296202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.306789] alg: No test for adler32 (adler32-zlib) [ 53.370522] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 53.761959] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 55.408195] Key type lgssc registered [ 56.837828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 68.414056] hrtimer: interrupt took 8044997 ns [ 72.371678] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 73.480061] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 80.109349] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 87.267303] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.153896] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.244632] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 118.280105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 118.359338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 118.390570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 119.780169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 119.828581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 119.952866] Lustre: lustre-MDT0000: new disk, initializing [ 120.080756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 120.097932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 123.876541] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 135.901270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.001126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 136.067882] Lustre: 6491:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 136.117318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 136.124549] Lustre: Skipped 1 previous similar message [ 136.260982] Lustre: lustre-MDT0001: new disk, initializing [ 136.359341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 136.403352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 136.412442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 139.947322] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 144.716555] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 153.641464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 153.726483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 153.980575] Lustre: lustre-OST0000: new disk, initializing [ 153.985766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 154.062367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.989620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 157.997039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 158.092312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 159.132770] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.663931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 172.746494] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 172.832657] Lustre: lustre-OST0001: new disk, initializing [ 172.835538] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 172.907160] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 177.850871] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.202342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 178.207663] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 178.295755] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 187.985574] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 193.514317] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 203.019616] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing check_logdir /tmp/testlogs/ [ 206.764374] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing yml_node [ 210.858605] Lustre: DEBUG MARKER: Client: 2.16.61.41 [ 213.178780] Lustre: DEBUG MARKER: MDS: 2.16.61.41 [ 215.586403] Lustre: DEBUG MARKER: OSS: 2.16.61.41 [ 217.251857] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 22:49:55 EST 2025 [ 236.212417] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 237.809677] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 254.434301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 254.438466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.485402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 254.946147] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 260.066188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 260.068697] Lustre: Skipped 5 previous similar messages [ 265.186340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 267.232142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 267.358919] LustreError: 12195:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 267.423239] Lustre: server umount lustre-MDT0000 complete [ 270.625725] LustreError: 7434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351451 with bad export cookie 4738059541436691922 [ 270.628355] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 270.640970] LustreError: 7434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 275.428534] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 275.430085] LustreError: 6497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 275.430469] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 275.430473] Lustre: Skipped 3 previous similar messages [ 275.466435] Lustre: Skipped 3 previous similar messages [ 275.487451] LustreError: 6497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 276.824363] LustreError: 12398:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 276.828048] LustreError: 12398:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 276.922728] Lustre: server umount lustre-MDT0001 complete [ 280.334628] LustreError: 12600:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 280.337471] LustreError: 12600:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 280.405959] Lustre: server umount lustre-OST0000 complete [ 283.615842] LustreError: 12800:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 283.621110] LustreError: 12800:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 283.732358] Lustre: server umount lustre-OST0001 complete [ 289.465753] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 297.415950] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 306.431564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 312.135662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 319.044421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 325.694382] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 332.565709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 332.642053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 332.879278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 332.915693] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 332.970477] Lustre: lustre-MDT0000: new disk, initializing [ 333.064443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 333.095262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 336.153958] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 345.772540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 345.837273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 345.901320] Lustre: 14985:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 345.963946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 345.979217] Lustre: Skipped 1 previous similar message [ 346.066659] Lustre: lustre-MDT0001: new disk, initializing [ 346.170598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 346.208867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 346.220064] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 349.516235] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 355.621148] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 358.504329] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 364.994348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 365.081082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 365.332923] Lustre: lustre-OST0000: new disk, initializing [ 365.336143] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 365.440891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 367.233763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 367.251723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 367.437340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 370.666293] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 377.148056] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 380.730745] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 380.955650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 384.235317] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 384.435166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 388.070540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 388.076296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 388.079477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 388.088327] Lustre: Skipped 1 previous similar message [ 388.101117] Lustre: Skipped 2 previous similar messages [ 398.307177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 398.309676] Lustre: Skipped 3 previous similar messages [ 400.352194] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 400.391337] LustreError: 17927:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 400.405642] LustreError: 17927:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 400.519776] Lustre: server umount lustre-OST0000 complete [ 407.529641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 407.534530] LustreError: Skipped 1 previous similar message [ 407.541190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 409.891667] LustreError: 18130:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 409.896817] LustreError: 18130:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 409.968442] Lustre: server umount lustre-MDT0000 complete [ 413.044115] LustreError: 14975:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351593 with bad export cookie 4738059541436693455 [ 413.044621] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 413.052674] LustreError: 14975:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 413.487471] Lustre: server umount lustre-MDT0001 complete [ 417.082712] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 22:53:14 (1763351594) [ 419.270925] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 420.919882] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 22:53:18 (1763351598) [ 426.462979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 427.024057] LustreError: 19001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 427.107480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 430.229232] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 432.098026] LustreError: 19001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 435.841645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 438.934293] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 443.250417] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 446.262155] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 452.248939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 456.962402] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 461.096653] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 468.963454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 468.966259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 468.974617] Lustre: Skipped 1 previous similar message [ 468.977997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 468.980941] Lustre: Skipped 3 previous similar messages [ 472.146161] LustreError: 21245:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 472.151147] LustreError: 21245:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 472.301796] Lustre: server umount lustre-OST0000 complete [ 481.661789] Lustre: server umount lustre-MDT0000 complete [ 484.594033] LustreError: 18980:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351664 with bad export cookie 4738059541436694337 [ 484.602251] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 484.612797] LustreError: 18980:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 484.988208] Lustre: server umount lustre-MDT0001 complete [ 493.985690] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 496.480675] Key type lgssc unregistered [ 496.810246] LNet: 22182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 496.819268] LNetError: 22182:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 497.830226] LNet: Removed LNI 192.168.202.129@tcp [ 498.843276] Key type .llcrypt unregistered [ 498.846611] Key type ._llcrypt unregistered [ 517.513527] Key type ._llcrypt registered [ 517.515316] Key type .llcrypt registered [ 517.612575] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 518.501644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 518.581638] alg: No test for adler32 (adler32-zlib) [ 519.634594] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 519.835689] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 521.528204] Key type lgssc registered [ 522.673172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 529.737439] loop: module loaded [ 529.820965] loop0: detected capacity change from 0 to 204800000 [ 530.007214] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 530.014724] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 530.183121] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 530.232517] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 535.341888] loop0: detected capacity change from 0 to 204800000 [ 535.499539] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 535.509204] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 535.523386] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 535.659630] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 540.273611] loop0: detected capacity change from 0 to 204800000 [ 545.343063] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 545.369139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 545.376432] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 546.549080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 546.580721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 546.634295] Lustre: lustre-MDT0000: new disk, initializing [ 546.732100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 546.753787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 550.446632] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 558.467989] loop1: detected capacity change from 0 to 204800000 [ 564.447391] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 564.515067] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 564.696455] Lustre: lustre-OST07c6: new disk, initializing [ 564.699748] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 564.701729] Lustre: Skipped 1 previous similar message [ 564.770340] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 570.002915] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 572.547736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 572.560231] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 572.611356] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 574.184752] Lustre: 24319:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 574.189939] Lustre: 24319:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 574.195318] Lustre: 24319:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 574.202360] Lustre: 24319:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 574.208763] Lustre: 24319:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 574.212583] Lustre: 24319:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 590.394196] Lustre: 24318:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 590.409687] Lustre: 24318:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 590.416294] Lustre: 24318:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 590.420890] Lustre: 24318:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 590.425368] Lustre: 24318:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 590.430442] Lustre: 24318:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 595.133784] Lustre: 24318:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 595.139388] Lustre: 24318:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 595.143504] Lustre: 24318:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 595.148205] Lustre: 24318:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 595.153233] Lustre: 24318:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 595.161151] Lustre: 24318:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 595.164459] Lustre: 24318:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 595.168151] Lustre: 24318:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 595.178165] Lustre: 24318:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 595.182922] Lustre: 24318:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 595.187143] Lustre: 24318:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 595.195796] Lustre: 24318:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 598.358898] Lustre: 25988:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 598.366903] Lustre: 25988:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 598.373281] Lustre: 25988:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 598.378383] Lustre: 25988:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 598.383434] Lustre: 25988:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 598.388345] Lustre: 25988:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 598.391519] Lustre: 25988:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 598.395454] Lustre: 25988:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 598.399045] Lustre: 25988:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 598.403435] Lustre: 25988:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 598.407560] Lustre: 25988:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 598.412621] Lustre: 25988:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.813218] Lustre: 25988:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 602.827797] Lustre: 25988:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 602.845105] Lustre: 25988:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 602.856671] Lustre: 25988:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.860169] Lustre: 25988:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 602.865899] Lustre: 25988:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.872050] Lustre: 25988:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 602.877585] Lustre: 25988:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.880792] Lustre: 25988:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 602.889328] Lustre: 25988:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 602.896050] Lustre: 25988:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 602.904459] Lustre: 25988:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 613.896168] Lustre: 24318:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 613.904396] Lustre: 24318:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 5 previous similar messages [ 613.909205] Lustre: 24318:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 613.912432] Lustre: 24318:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 613.917851] Lustre: 24318:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 613.921853] Lustre: 24318:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 613.926162] Lustre: 24318:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 613.930994] Lustre: 24318:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 613.935484] Lustre: 24318:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 613.938961] Lustre: 24318:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 613.943847] Lustre: 24318:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 613.948538] Lustre: 24318:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 664.548132] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 664.562658] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 666.558054] LustreError: 26507:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 666.665320] Lustre: server umount lustre-OST07c6 complete [ 673.161630] LustreError: 27009:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 673.164700] LustreError: 27009:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 673.350217] Lustre: server umount lustre-MDT0000 complete [ 685.675921] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 688.078366] Key type lgssc unregistered [ 688.298700] LNet: 27843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.310981] LNetError: 27843:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 688.329640] LNet: Removed LNI 192.168.202.129@tcp [ 689.022387] Key type .llcrypt unregistered [ 689.023891] Key type ._llcrypt unregistered [ 701.830979] Key type ._llcrypt registered [ 701.834253] Key type .llcrypt registered [ 701.906135] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 702.349292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 702.357223] alg: No test for adler32 (adler32-zlib) [ 703.260760] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 703.399507] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 705.024213] Key type lgssc registered [ 705.674750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 711.464033] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 711.491114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.915772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.830351] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 718.310479] LustreError: 29074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 718.319833] LustreError: 29074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 722.426595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 722.741941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 726.063702] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 730.733398] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 733.691665] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 740.099733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 740.387852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 745.245703] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 749.680316] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 758.831925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 770.027258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 770.027840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 770.035490] LustreError: Skipped 1 previous similar message [ 770.058952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 774.115057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 774.115584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 774.129248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.238731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 779.241364] Lustre: Skipped 2 previous similar messages [ 784.354174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 784.356057] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 784.361159] Lustre: Skipped 1 previous similar message [ 784.402241] LustreError: 31543:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 784.480241] Lustre: server umount lustre-OST0000 complete [ 789.474665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 789.475099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 789.487077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 789.500318] Lustre: Skipped 1 previous similar message [ 793.303755] LustreError: 31745:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 793.307093] LustreError: 31745:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 793.354964] Lustre: server umount lustre-MDT0000 complete [ 794.597197] LustreError: 29069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 794.604628] LustreError: 29069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 795.811243] LustreError: 29053:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351976 with bad export cookie 14849018810160712556 [ 795.818605] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 795.825309] LustreError: 29053:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 795.939475] LustreError: 31949:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 795.946666] LustreError: 31949:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 796.086991] Lustre: server umount lustre-MDT0001 complete [ 802.030186] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 803.988948] Key type lgssc unregistered [ 804.215782] LNet: 32480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.222319] LNetError: 32480:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.287382] LNet: Removed LNI 192.168.202.129@tcp [ 805.941118] Key type .llcrypt unregistered [ 805.943769] Key type ._llcrypt unregistered [ 817.988992] Key type ._llcrypt registered [ 817.990223] Key type .llcrypt registered [ 818.071320] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 818.602359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 818.677130] alg: No test for adler32 (adler32-zlib) [ 819.659361] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 819.836389] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 821.473112] Key type lgssc registered [ 822.336680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 828.759121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 828.778977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 830.111210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 833.011968] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 838.952403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 839.238433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 841.795660] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 845.584769] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 848.152850] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 852.958979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 853.212365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 855.281376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 857.157909] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 860.584604] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 870.884317] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 870.887942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 870.898705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 874.977650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 874.980799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 874.984877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 874.999401] Lustre: Skipped 1 previous similar message [ 880.100364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 880.103016] Lustre: Skipped 1 previous similar message [ 885.216135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 885.220538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 885.231286] Lustre: Skipped 1 previous similar message [ 885.253392] LustreError: 35956:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 885.320248] Lustre: server umount lustre-OST0000 complete [ 890.339959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 890.341580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 890.347893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 890.359308] Lustre: Skipped 1 previous similar message [ 893.655666] LustreError: 36171:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 893.658547] LustreError: 36171:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 893.702534] Lustre: server umount lustre-MDT0000 complete [ 895.460561] LustreError: 34494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 895.468830] LustreError: 34494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 896.180809] LustreError: 35327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352076 with bad export cookie 6016427486051140891 [ 896.182638] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 896.194689] LustreError: 35327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 896.313878] LustreError: 36373:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 896.317796] LustreError: 36373:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 896.441324] Lustre: server umount lustre-MDT0001 complete [ 902.456703] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 903.956347] Key type lgssc unregistered [ 904.147074] LNet: 36906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.153275] LNetError: 36906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.164897] LNet: Removed LNI 192.168.202.129@tcp [ 904.627124] Key type .llcrypt unregistered [ 904.628914] Key type ._llcrypt unregistered [ 917.210401] Key type ._llcrypt registered [ 917.211666] Key type .llcrypt registered [ 917.258936] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 917.693808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 917.750940] alg: No test for adler32 (adler32-zlib) [ 918.661418] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 918.783385] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 920.394721] Key type lgssc registered [ 920.984928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 925.559408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 925.572594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 926.846453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 928.890319] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 934.781723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 935.014538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 937.482642] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 941.542730] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 943.810579] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 947.948891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 948.136336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 951.644248] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 952.239740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 954.805356] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 965.606625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 965.609278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 965.609556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 965.613182] Lustre: Skipped 2 previous similar messages [ 966.380173] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.29@tcp (stopping) [ 966.384166] Lustre: Skipped 2 previous similar messages [ 968.601095] LustreError: 40524:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 968.660992] Lustre: server umount lustre-MDT0000 complete [ 970.724304] LustreError: 38281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 970.731807] LustreError: 38281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 971.003472] LustreError: 38260:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352151 with bad export cookie 588165770024144139 [ 971.015021] LustreError: 38260:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 971.018449] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 971.484148] LustreError: 38276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.500678] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.29@tcp (stopping) [ 975.844734] LustreError: 38276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 975.846494] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.871489] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 977.166725] LustreError: 40725:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 977.169980] LustreError: 40725:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 977.251984] Lustre: server umount lustre-MDT0001 complete [ 996.412060] LustreError: 40949:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 996.414442] LustreError: 40949:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 996.499390] Lustre: server umount lustre-OST0000 complete [ 1005.474326] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 1006.829297] Key type lgssc unregistered [ 1007.018862] LNet: 41786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.025230] LNetError: 41786:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.042950] LNet: Removed LNI 192.168.202.129@tcp [ 1007.501788] Key type .llcrypt unregistered [ 1007.503469] Key type ._llcrypt unregistered [ 1021.623648] Key type ._llcrypt registered [ 1021.625265] Key type .llcrypt registered [ 1021.704495] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 1022.360739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1022.370019] alg: No test for adler32 (adler32-zlib) [ 1023.374148] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1023.603856] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 1025.248502] Key type lgssc registered [ 1025.978758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1031.466440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1031.484681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.811286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.183693] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1039.913220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1040.106945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1042.047729] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1044.897699] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1046.891164] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1050.973173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1051.245995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1054.384911] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1057.384238] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1058.413585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1074.145244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1074.148674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.155889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1075.436209] Lustre: lustre-OST0000: Not available for connect from 192.168.202.29@tcp (stopping) [ 1076.194692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1076.204967] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.618833] LustreError: 45427:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1077.688546] Lustre: server umount lustre-OST0000 complete [ 1096.034079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1096.036435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1096.041623] Lustre: Skipped 1 previous similar message [ 1096.046683] Lustre: Skipped 3 previous similar messages [ 1101.281749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1101.285089] Lustre: Skipped 1 previous similar message [ 1106.400271] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1106.401709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1106.407225] Lustre: Skipped 1 previous similar message [ 1106.443526] LustreError: 45806:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1106.447686] LustreError: 45806:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1106.481190] Lustre: server umount lustre-MDT0000 complete [ 1108.160177] LustreError: 43144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352288 with bad export cookie 3360122937379537824 [ 1108.161807] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1108.165311] LustreError: 43144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1114.330451] LustreError: 46008:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1114.334906] LustreError: 46008:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1114.416981] Lustre: server umount lustre-MDT0001 complete [ 1119.360859] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 1120.741931] Key type lgssc unregistered [ 1120.888431] LNet: 46541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1120.892446] LNetError: 46541:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1120.903814] LNet: Removed LNI 192.168.202.129@tcp [ 1121.303254] Key type .llcrypt unregistered [ 1121.304593] Key type ._llcrypt unregistered [ 1132.246102] Key type ._llcrypt registered [ 1132.247430] Key type .llcrypt registered [ 1132.297326] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 1132.893440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1132.906393] alg: No test for adler32 (adler32-zlib) [ 1133.797104] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1133.914046] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 1135.520170] Key type lgssc registered [ 1136.025166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1139.709216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1139.723882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1141.039950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1143.176884] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1148.239729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1148.470258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1150.356194] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1153.150157] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1154.970148] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1158.394823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1158.597282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1161.130977] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1163.913606] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1165.803781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1176.769564] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1178.345495] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1189.392872] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-11-1623h06m07s [ 1190.116089] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1193.443825] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.445486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.451440] Lustre: Skipped 1 previous similar message [ 1194.468522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1194.470776] Lustre: Skipped 4 previous similar messages [ 1196.263615] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.29@tcp (stopping) [ 1197.383752] LustreError: 50507:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1197.422324] Lustre: server umount lustre-MDT0000 complete [ 1198.933104] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1200.612467] LustreError: 50072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1200.619324] LustreError: 50072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1201.951932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1202.035184] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1202.174363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1202.206460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1203.182624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1204.006559] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1205.562882] Lustre: DEBUG MARKER: Wait for df (36006) ... [ 1206.282611] Lustre: DEBUG MARKER: done [ 1208.293978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.304397] Lustre: Skipped 1 previous similar message [ 1208.307437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1208.309705] Lustre: Skipped 2 previous similar messages [ 1213.413446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1213.416236] Lustre: Skipped 1 previous similar message [ 1213.747342] LustreError: 51554:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1213.754767] LustreError: 51554:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1213.807992] Lustre: server umount lustre-OST0000 complete [ 1218.529955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1218.534974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1218.543625] Lustre: Skipped 2 previous similar messages [ 1221.512435] LustreError: 51758:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1221.515726] LustreError: 51758:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1221.552154] Lustre: server umount lustre-MDT0000 complete [ 1223.248622] LustreError: 50238:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352403 with bad export cookie 6998562757377744126 [ 1223.250059] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1223.254095] LustreError: 50238:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1223.403769] Lustre: server umount lustre-MDT0001 complete [ 1227.917796] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 1229.255604] Key type lgssc unregistered [ 1229.438283] LNet: 52491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1229.441377] LNetError: 52491:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1229.452092] LNet: Removed LNI 192.168.202.129@tcp [ 1229.879211] Key type .llcrypt unregistered [ 1229.881130] Key type ._llcrypt unregistered [ 1245.965861] Key type ._llcrypt registered [ 1245.967424] Key type .llcrypt registered [ 1246.028871] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 1253.998573] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 1254.478834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1254.562161] alg: No test for adler32 (adler32-zlib) [ 1255.491581] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1255.625483] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 1257.256544] Key type lgssc registered [ 1257.842912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1262.057192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1265.283783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1268.031633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1270.751069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1273.648704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1273.678929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1273.688094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1274.796224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1274.814173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1274.857675] Lustre: lustre-MDT0000: new disk, initializing [ 1274.897517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1274.907092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1276.486812] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1281.387247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1281.413887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1281.454487] Lustre: 56413:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1281.483655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1281.486509] Lustre: Skipped 1 previous similar message [ 1281.526624] Lustre: lustre-MDT0001: new disk, initializing [ 1281.573475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1281.590531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1281.594545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1283.139201] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1286.789117] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1288.274723] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1291.066207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1291.097762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1291.210253] Lustre: lustre-OST0000: new disk, initializing [ 1291.213238] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1291.253703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1293.522288] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1294.925835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1294.931925] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1294.974796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1297.303963] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1299.005592] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1299.095945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1300.655943] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1300.744626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1305.571359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1305.571464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.573660] LustreError: Skipped 1 previous similar message [ 1305.578327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1305.580167] Lustre: Skipped 1 previous similar message [ 1307.569085] LustreError: 59349:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1307.622448] Lustre: server umount lustre-OST0000 complete [ 1310.691276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1310.699419] Lustre: Skipped 1 previous similar message [ 1310.702215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1315.404666] LustreError: 59551:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1315.408015] LustreError: 59551:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1315.457848] Lustre: server umount lustre-MDT0000 complete [ 1317.492707] LustreError: 56404:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352497 with bad export cookie 11643004378488533005 [ 1317.494631] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1317.497546] LustreError: 56404:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1317.577026] LustreError: 59754:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1317.584162] LustreError: 59754:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1317.666919] Lustre: server umount lustre-MDT0001 complete [ 1321.648776] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:08:20 (1763352500) [ 1324.765504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1324.971163] LustreError: 60425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1324.980302] LustreError: 60425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1325.011034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1326.617401] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1330.148462] LustreError: 60425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1330.155147] LustreError: 60425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1330.567074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1330.744636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1332.735321] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1335.245539] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1337.101324] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1340.579563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1340.748465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1343.588677] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1346.380664] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1354.346903] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-11-1623h08m53s [ 1355.011609] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1358.455433] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1359.623801] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1359.640517] LustreError: 60420:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1380.087664] Lustre: lustre-MDT0000: Client 6096bac3-d935-4ecd-9474-8a0a9600f867 (at 192.168.202.29@tcp) reconnecting [ 1399.688139] LustreError: 60420:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1400.725916] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1401.776554] Lustre: DEBUG MARKER: done [ 1405.925897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1405.930357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1405.936187] Lustre: Skipped 1 previous similar message [ 1405.939276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1405.942097] Lustre: Skipped 1 previous similar message [ 1409.916155] LustreError: 63426:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1409.921068] LustreError: 63426:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1409.974423] Lustre: server umount lustre-OST0000 complete [ 1412.577303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1412.580679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1412.586122] Lustre: Skipped 1 previous similar message [ 1412.588671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1412.591544] Lustre: Skipped 2 previous similar messages [ 1417.633737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1417.636713] Lustre: Skipped 4 previous similar messages [ 1417.801473] LustreError: 63628:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1417.803755] LustreError: 63628:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1417.842352] Lustre: server umount lustre-MDT0000 complete [ 1419.619821] LustreError: 60406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352600 with bad export cookie 11643004378488533887 [ 1419.623398] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1419.625377] LustreError: 60406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1419.769396] Lustre: server umount lustre-MDT0001 complete [ 1424.434346] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 1425.723572] Key type lgssc unregistered [ 1425.872919] LNet: 64360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1425.876914] LNetError: 64360:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1425.889445] LNet: Removed LNI 192.168.202.129@tcp [ 1426.268658] Key type .llcrypt unregistered [ 1426.270331] Key type ._llcrypt unregistered [ 1441.179898] Key type ._llcrypt registered [ 1441.181575] Key type .llcrypt registered [ 1441.234103] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 1448.554413] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 1449.035409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1449.067437] alg: No test for adler32 (adler32-zlib) [ 1449.972946] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1450.080723] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 1451.680216] Key type lgssc registered [ 1452.168116] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1456.335851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1459.260442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1462.102619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1465.694482] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1469.209811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1469.235389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1469.244240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.387277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1470.412537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1470.457942] Lustre: lustre-MDT0000: new disk, initializing [ 1470.509893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1470.521905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1472.371934] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1477.669199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1477.700619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1477.753622] Lustre: 68285:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1477.788341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1477.791196] Lustre: Skipped 1 previous similar message [ 1477.832821] Lustre: lustre-MDT0001: new disk, initializing [ 1477.898296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1477.919687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1477.924200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1479.968277] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1484.352330] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1486.252330] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1489.669045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1489.702872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1489.830295] Lustre: lustre-OST0000: new disk, initializing [ 1489.833718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1489.883724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1492.613820] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1496.557389] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1498.406390] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1498.629784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1498.634647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1498.684696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1499.578807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1501.458813] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1501.555022] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1503.715140] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1503.716119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1503.721316] LustreError: Skipped 1 previous similar message [ 1503.728117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1503.732497] Lustre: Skipped 1 previous similar message [ 1504.435018] LustreError: 71232:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1504.480981] Lustre: server umount lustre-OST0000 complete [ 1508.833710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1508.837319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1508.838901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1508.842938] Lustre: Skipped 2 previous similar messages [ 1512.146684] LustreError: 71434:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1512.150094] LustreError: 71434:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1512.201752] Lustre: server umount lustre-MDT0000 complete [ 1514.039629] LustreError: 68276:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352694 with bad export cookie 3226845803182928144 [ 1514.044647] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1514.045186] LustreError: 68276:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1514.125388] LustreError: 71635:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1514.127848] LustreError: 71635:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1514.197658] Lustre: server umount lustre-MDT0001 complete [ 1518.195276] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 23:11:36 (1763352696) [ 1518.877488] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1519.645896] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 23:11:38 (1763352698) [ 1524.966327] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 1527.203568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1527.457154] LustreError: 72682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1527.468227] LustreError: 72682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1527.503978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1528.019612] Lustre: Failing over lustre-MDT0000 [ 1528.026101] LustreError: 72710:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1528.032973] LustreError: 72710:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1528.043771] LustreError: 72710:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1528.141613] LustreError: 72761:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1528.144439] LustreError: 72761:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1528.224576] Lustre: server umount lustre-MDT0000 complete [ 1531.131146] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:11:49 (1763352709) [ 1534.148286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1534.348299] LustreError: 73330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.375447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1536.134484] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1539.555074] LustreError: 73330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1540.300150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1542.441737] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1544.896729] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1546.594241] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1549.983235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1550.201078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1550.203997] Lustre: Skipped 1 previous similar message [ 1553.073695] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1555.534653] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1567.947236] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1570.786898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1570.788181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1570.792107] Lustre: Skipped 2 previous similar messages [ 1570.794058] Lustre: Skipped 2 previous similar messages [ 1575.905841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1576.711735] LustreError: 75697:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1576.717919] LustreError: 75697:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1576.763579] Lustre: server umount lustre-MDT0000 complete [ 1578.487748] LustreError: 73310:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352758 with bad export cookie 3226845803182929285 [ 1578.492352] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1578.493046] LustreError: 73310:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1578.647114] Lustre: server umount lustre-MDT0001 complete [ 1580.265761] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1582.444768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1586.423276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1588.193323] LustreError: 66498:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c3ce1916 x1849009247415424/t0(0) o250->MGC192.168.202.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1588.388765] LustreError: 74936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1588.434650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1588.474080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1590.355050] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1593.830939] LustreError: 74936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1593.832788] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1594.772379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1596.907528] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1597.537131] Lustre: 66500:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763352761/real 1763352761] req@00000000bc922c02 x1849009247415040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763352777 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1599.584427] Lustre: 66500:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763352763/real 1763352763] req@000000002bf25fea x1849009247415168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763352779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1599.744881] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1599.974327] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1601.001697] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1601.369589] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1602.016229] Lustre: 66501:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763352766/real 1763352766] req@00000000fbe779e0 x1849009247415296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763352782 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1604.048206] Lustre: 66501:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763352768/real 1763352768] req@0000000000b76296 x1849009247415552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763352784 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1605.089970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1605.093707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1605.099799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1605.102523] Lustre: Skipped 2 previous similar messages [ 1609.860553] LustreError: 78155:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1609.862774] LustreError: 78155:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1609.906579] Lustre: server umount lustre-MDT0000 complete [ 1611.235366] LustreError: 76754:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1611.242523] LustreError: 76754:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1611.670043] LustreError: 73310:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352792 with bad export cookie 3226845803182931175 [ 1611.675154] LustreError: 73310:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1611.675854] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1611.818391] Lustre: server umount lustre-MDT0001 complete [ 1614.782359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1619.410650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1622.499180] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2cc80f2953adc243 [ 1622.503428] Lustre: MGC192.168.202.129@tcp: Connection restored to 0@lo (at 0@lo) [ 1622.703949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1622.708472] Lustre: Skipped 1 previous similar message [ 1622.744086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1624.552353] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1628.131515] LustreError: 74936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1628.133546] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.139550] LustreError: 74936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1628.152215] Lustre: Skipped 2 previous similar messages [ 1628.355181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1630.563940] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1631.712348] Lustre: 66502:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763352796/real 1763352796] req@00000000999d8a65 x1849009247435776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763352812 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1633.138528] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1633.767201] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1634.608083] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1638.881350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1638.885141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1638.887631] Lustre: Skipped 3 previous similar messages [ 1643.207525] LustreError: 80612:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1643.210248] LustreError: 80612:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1643.248790] Lustre: server umount lustre-MDT0000 complete [ 1644.963924] LustreError: 73310:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352825 with bad export cookie 3226845803182932547 [ 1644.968515] LustreError: 73310:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1644.968957] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1645.121584] Lustre: server umount lustre-MDT0001 complete [ 1647.929293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1649.436406] Lustre: DEBUG MARKER: files compared the same [ 1660.596334] Lustre: server umount lustre-OST0000 complete [ 1667.318176] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 1668.706972] Key type lgssc unregistered [ 1668.883725] LNet: 82243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.887710] LNetError: 82243:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.897806] LNet: Removed LNI 192.168.202.129@tcp [ 1669.364879] Key type .llcrypt unregistered [ 1669.366268] Key type ._llcrypt unregistered [ 1679.969526] Key type ._llcrypt registered [ 1679.971030] Key type .llcrypt registered [ 1680.042443] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 1680.605795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1680.623056] alg: No test for adler32 (adler32-zlib) [ 1681.514366] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1681.626435] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 1683.225768] Key type lgssc registered [ 1683.695111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1686.941988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1686.950543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1688.168146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.848356] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug +malloc all [ 1693.558043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1693.717456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1695.429782] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug +malloc all [ 1697.711581] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1699.222468] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1702.106306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1702.244493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1704.678459] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug +malloc all [ 1706.962510] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1713.509414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1723.873955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.874186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1723.874443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1723.881322] Lustre: Skipped 1 previous similar message [ 1726.382911] LustreError: 85888:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1726.424365] Lustre: server umount lustre-OST0000 complete [ 1728.994803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1729.001767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1729.004596] Lustre: Skipped 2 previous similar messages [ 1734.114087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.118439] Lustre: Skipped 1 previous similar message [ 1739.232732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1739.236773] Lustre: Skipped 1 previous similar message [ 1742.304163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1742.343327] LustreError: 86091:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1742.345671] LustreError: 86091:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1742.380976] Lustre: server umount lustre-MDT0000 complete [ 1744.100727] LustreError: 86092:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352924 with bad export cookie 13151447002042048111 [ 1744.102441] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1744.105158] LustreError: 86092:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1744.189670] LustreError: 86295:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1744.193026] LustreError: 86295:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1744.257969] Lustre: server umount lustre-MDT0001 complete [ 1748.743571] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 1750.057606] Key type lgssc unregistered [ 1750.210717] LNet: 86827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1750.214437] LNetError: 86827:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1750.223415] LNet: Removed LNI 192.168.202.129@tcp [ 1750.577356] Key type .llcrypt unregistered [ 1750.579117] Key type ._llcrypt unregistered [ 1761.644953] Key type ._llcrypt registered [ 1761.646885] Key type .llcrypt registered [ 1761.703701] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 1762.194223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1762.239159] alg: No test for adler32 (adler32-zlib) [ 1763.161134] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1763.289225] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 1764.912207] Key type lgssc registered [ 1765.484862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1769.296252] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1769.305606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1784.800374] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1789.920668] LustreError: 88173:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.129@tcp: failed processing log, type 4: rc = -110 [ 1816.544277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1819.185271] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1821.518732] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1824.886565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1824.965915] Lustre: *** cfs_fail_loc=706, val=0*** [ 1825.171730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.204868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1826.855694] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1830.370207] LustreError: 88200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1830.375313] LustreError: 88200:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1830.706369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1830.903684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1832.685782] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1835.209571] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1836.681914] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1840.608994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1840.611674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.617298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1841.121929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1841.125577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.132381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1842.146754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1842.150601] Lustre: Skipped 1 previous similar message [ 1843.631686] LustreError: 90494:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1843.686161] Lustre: server umount lustre-OST0000 complete [ 1846.241462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1846.245122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.251137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1851.524303] LustreError: 90699:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1851.527077] LustreError: 90699:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1851.557422] Lustre: server umount lustre-MDT0000 complete [ 1852.385381] LustreError: 89149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1852.390567] LustreError: 89149:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1853.084409] LustreError: 88180:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353033 with bad export cookie 6659482316963775335 [ 1853.086930] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1853.089891] LustreError: 88180:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1853.165528] LustreError: 90901:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1853.168253] LustreError: 90901:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1853.229836] Lustre: server umount lustre-MDT0001 complete [ 1857.714121] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 1858.969379] Key type lgssc unregistered [ 1859.123663] LNet: 91432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1859.127298] LNetError: 91432:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1859.134424] LNet: Removed LNI 192.168.202.129@tcp [ 1859.496243] Key type .llcrypt unregistered [ 1859.497782] Key type ._llcrypt unregistered [ 1871.135798] Key type ._llcrypt registered [ 1871.138115] Key type .llcrypt registered [ 1871.186757] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 1871.569952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1871.595260] alg: No test for adler32 (adler32-zlib) [ 1872.471724] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 1872.581579] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 1874.176472] Key type lgssc registered [ 1874.655448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1877.915711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1877.925080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.560369] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1883.951850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1884.241249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1885.982370] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1889.249291] LustreError: 93475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1889.256622] LustreError: 93475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1889.803293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1889.967405] LustreError: 93475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1889.977466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1892.410512] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1894.051432] LustreError: 93497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1894.060843] LustreError: 93497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1896.288974] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1896.295726] Lustre: MGS: Client 3b259fea-079b-4643-bcf1-ef5e0a2bf1be (at 0@lo) reconnecting [ 1896.432378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1896.433526] LustreError: 93490:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1896.490055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1898.332096] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1906.658769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1906.660699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1906.660820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1906.663494] Lustre: Skipped 1 previous similar message [ 1911.779203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1911.783364] Lustre: Skipped 2 previous similar messages [ 1912.688249] LustreError: 95437:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1912.735365] Lustre: server umount lustre-OST0000 complete [ 1916.898524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1916.898585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1916.905794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1916.908459] Lustre: Skipped 1 previous similar message [ 1920.392249] LustreError: 95641:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1920.394675] LustreError: 95641:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1920.404087] Lustre: server umount lustre-MDT0000 complete [ 1922.096157] LustreError: 95842:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1922.099114] LustreError: 95842:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1922.168440] Lustre: server umount lustre-MDT0001 complete [ 1923.790126] Lustre: server umount MGS complete [ 1928.705992] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 23:18:27 (1763353107) [ 1940.991370] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 1944.962861] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 1949.059719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1951.621648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1954.142743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1956.728636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1959.693392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1959.728636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1961.526315] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1966.322862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1966.351847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1966.473386] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1966.499122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1966.549379] Lustre: lustre-MDT0001: new disk, initializing [ 1966.590755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1966.604248] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1966.607913] Lustre: 100559:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1966.611926] LustreError: 100559:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [ 1968.358773] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1973.335980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1973.369329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1973.486453] Lustre: lustre-OST0000: new disk, initializing [ 1973.489346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1973.521694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.805588] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1979.425174] Lustre: 101820:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000580a91f8 x1849009689888256/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763353175 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1979.511996] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1982.572271] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1982.578200] Lustre: MGS: Client a7672151-7cbc-4555-9114-44780ba9f66c (at 0@lo) reconnecting [ 1982.597947] Lustre: 99845:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1982.608490] LustreError: 99845:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1982.623926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1982.668922] Lustre: lustre-MDT0000: new disk, initializing [ 1982.733426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1982.747461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1983.779418] Lustre: 102270:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000005bba94d3 x1849009689893504/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1763353180 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1983.841710] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1984.476103] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 1988.518161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1988.521391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1988.905427] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2009.057112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2009.060184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.066127] Lustre: Skipped 1 previous similar message [ 2009.068527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2016.737500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2016.740289] Lustre: Skipped 4 previous similar messages [ 2022.880812] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2022.906206] LustreError: 102790:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2022.908324] LustreError: 102790:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2022.979738] Lustre: server umount lustre-OST0000 complete [ 2026.977852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2026.978173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.981430] LustreError: Skipped 1 previous similar message [ 2026.987155] Lustre: Skipped 1 previous similar message [ 2026.988481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2026.995624] Lustre: Skipped 3 previous similar messages [ 2030.915452] LustreError: 102994:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2030.917893] LustreError: 102994:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2030.928051] Lustre: server umount lustre-MDT0000 complete [ 2032.097711] LustreError: 100530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2032.104269] LustreError: 100530:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2038.671270] Lustre: server umount lustre-MDT0001 complete [ 2040.213918] LustreError: 103399:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2040.216871] LustreError: 103399:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2040.252919] Lustre: server umount MGS complete [ 2044.665914] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 23:20:23 (1763353223) [ 2051.329838] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 2052.470898] Key type lgssc unregistered [ 2052.602440] LNet: 104733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.605150] LNetError: 104733:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2052.612460] LNet: Removed LNI 192.168.202.129@tcp [ 2052.953933] Key type .llcrypt unregistered [ 2052.955462] Key type ._llcrypt unregistered [ 2060.716083] Key type ._llcrypt registered [ 2060.717554] Key type .llcrypt registered [ 2060.762095] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 2061.196992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2061.265676] alg: No test for adler32 (adler32-zlib) [ 2062.163442] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 2062.273900] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 2063.872245] Key type lgssc registered [ 2064.391806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2067.922800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2067.932580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2069.169833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.929950] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2074.633431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2074.812991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2076.493319] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2078.626260] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2080.041856] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2082.973921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2083.125471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2085.340125] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2087.426097] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2094.383036] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2104.806937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.807593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2104.808032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2104.812488] Lustre: Skipped 1 previous similar message [ 2106.932821] LustreError: 108178:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2106.988787] Lustre: server umount lustre-OST0000 complete [ 2109.921859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2109.922653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2109.927403] Lustre: Skipped 1 previous similar message [ 2109.930103] Lustre: Skipped 1 previous similar message [ 2114.632541] LustreError: 108380:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2114.635272] LustreError: 108380:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2114.671235] Lustre: server umount lustre-MDT0000 complete [ 2115.041690] LustreError: 107077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2115.048587] LustreError: 107077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2116.330824] LustreError: 105892:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353296 with bad export cookie 10903258003689942869 [ 2116.335316] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2116.336243] LustreError: 105892:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2116.417579] LustreError: 108581:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2116.420465] LustreError: 108581:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2116.502246] Lustre: server umount lustre-MDT0001 complete [ 2120.653723] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 2121.806047] Key type lgssc unregistered [ 2121.940516] LNet: 109113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2121.944061] LNetError: 109113:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2121.953421] LNet: Removed LNI 192.168.202.129@tcp [ 2122.278760] Key type .llcrypt unregistered [ 2122.280213] Key type ._llcrypt unregistered [ 2129.289376] Key type ._llcrypt registered [ 2129.290898] Key type .llcrypt registered [ 2129.336886] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 2129.703112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2129.725608] alg: No test for adler32 (adler32-zlib) [ 2130.611795] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 2130.718786] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 2132.312243] Key type lgssc registered [ 2132.786506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2135.333097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2135.333525] LustreError: 110152:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2135.335704] Lustre: Skipped 1 previous similar message [ 2135.335719] LustreError: 110153:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2135.347678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.348842] LustreError: 110152:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 2135.357708] LustreError: 110152:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2135.364153] LustreError: 110152:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2136.559483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.547059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2139.695487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2141.330443] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2143.488320] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing lsmod [ 2144.747341] LustreError: 111327:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2144.748790] LustreError: 111329:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2144.753139] LustreError: 111327:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 2144.757188] LustreError: 111327:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2144.763953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2144.764279] LustreError: 111327:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2144.925958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.960527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2149.861668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2149.865055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.871196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2150.883665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2150.883706] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2150.887081] Lustre: Skipped 1 previous similar message [ 2150.893611] Lustre: Skipped 1 previous similar message [ 2152.257908] LustreError: 111522:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2152.287676] Lustre: server umount lustre-MDT0000 complete [ 2153.683355] LustreError: 110167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353334 with bad export cookie 10107095543544136656 [ 2153.686428] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2153.688863] LustreError: 110167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2154.976246] LustreError: 110625:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ae2ba607 x1849009961451776/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2154.987947] LustreError: 111726:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2154.990493] LustreError: 111726:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2155.040585] Lustre: server umount lustre-MDT0001 complete [ 2166.644778] LustreError: 111927:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2166.647278] LustreError: 111927:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2166.693280] Lustre: server umount lustre-OST0000 complete [ 2170.026536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2170.230217] LustreError: 112404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.240025] LustreError: 112404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2170.264178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.830279] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2175.457384] LustreError: 112404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.570929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2175.739766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2177.410907] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2179.510660] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2180.921488] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2183.536841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2186.007770] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2188.233123] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2194.923120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 2201.056874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2201.059240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2201.064992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2205.156274] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2205.156907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2205.157089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2205.168208] Lustre: Skipped 1 previous similar message [ 2206.321116] LustreError: 114670:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2206.323991] LustreError: 114670:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2206.369173] Lustre: server umount lustre-OST0000 complete [ 2210.274577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2210.276401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2210.279121] Lustre: Skipped 1 previous similar message [ 2213.950508] LustreError: 114873:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2213.952615] LustreError: 114873:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2213.978137] Lustre: server umount lustre-MDT0000 complete [ 2215.360176] LustreError: 112385:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353395 with bad export cookie 10107095543544137559 [ 2215.363216] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2215.364765] LustreError: 112385:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2215.477860] Lustre: server umount lustre-MDT0001 complete [ 2219.403340] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 2220.525342] Key type lgssc unregistered [ 2220.674650] LNet: 115606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2220.678491] LNetError: 115606:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.685368] LNet: Removed LNI 192.168.202.129@tcp [ 2221.032824] Key type .llcrypt unregistered [ 2221.034379] Key type ._llcrypt unregistered [ 2231.350855] Key type ._llcrypt registered [ 2231.352322] Key type .llcrypt registered [ 2231.396348] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 2231.748823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2231.813218] alg: No test for adler32 (adler32-zlib) [ 2232.711431] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 2232.816347] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 2234.416188] Key type lgssc registered [ 2234.842831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2237.911666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2237.921740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.161300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2240.731294] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2244.349400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2244.519076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2246.074171] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2248.303614] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2249.891869] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2252.583909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2252.725316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2254.972417] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2257.007171] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2264.999411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 2269.487889] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2270.508828] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2274.273325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2274.276693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2274.283083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2275.301232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2275.301928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.302060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2277.747208] LustreError: 119346:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2277.792956] Lustre: server umount lustre-OST0000 complete [ 2280.418093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2280.418280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2280.418844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2280.418847] Lustre: Skipped 1 previous similar message [ 2280.429070] Lustre: Skipped 1 previous similar message [ 2285.376230] LustreError: 119551:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2285.379270] LustreError: 119551:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2285.410069] Lustre: server umount lustre-MDT0000 complete [ 2285.537536] LustreError: 116981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2285.542508] LustreError: 116981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2286.746137] LustreError: 119199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353467 with bad export cookie 16170953586316206060 [ 2286.750018] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2286.750609] LustreError: 119199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2286.821478] LustreError: 119755:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2286.823504] LustreError: 119755:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2286.879277] Lustre: server umount lustre-MDT0001 complete [ 2290.899760] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 2292.074655] Key type lgssc unregistered [ 2292.223512] LNet: 120285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2292.226396] LNetError: 120285:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2292.235414] LNet: Removed LNI 192.168.202.129@tcp [ 2292.573613] Key type .llcrypt unregistered [ 2292.575388] Key type ._llcrypt unregistered [ 2300.488923] Key type ._llcrypt registered [ 2300.491097] Key type .llcrypt registered [ 2300.543907] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 2300.902778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2300.951255] alg: No test for adler32 (adler32-zlib) [ 2301.836240] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 2301.932792] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 2303.528150] Key type lgssc registered [ 2303.970640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2307.234140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2307.242637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.455861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2309.943778] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2313.276042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2313.414797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2314.891893] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2316.995614] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2318.433727] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2321.208591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2321.359403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2323.799915] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2325.991972] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2332.651223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 2342.881828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.883816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2342.884199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2342.885803] Lustre: Skipped 1 previous similar message [ 2345.327086] LustreError: 123750:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2345.367260] Lustre: server umount lustre-OST0000 complete [ 2348.001699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2348.002877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2348.006458] Lustre: Skipped 1 previous similar message [ 2348.008941] Lustre: Skipped 1 previous similar message [ 2353.121210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2353.124060] Lustre: Skipped 1 previous similar message [ 2358.241217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2358.243948] Lustre: Skipped 1 previous similar message [ 2361.312256] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2361.353696] LustreError: 123953:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2361.355980] LustreError: 123953:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2361.388148] Lustre: server umount lustre-MDT0000 complete [ 2362.948732] LustreError: 123650:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353543 with bad export cookie 9319530013266799162 [ 2362.949663] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2362.953700] LustreError: 123650:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2363.026451] LustreError: 124157:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2363.029538] LustreError: 124157:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2363.095941] Lustre: server umount lustre-MDT0001 complete [ 2367.280048] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 2368.458153] Key type lgssc unregistered [ 2368.601839] LNet: 124688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2368.605525] LNetError: 124688:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2368.614410] LNet: Removed LNI 192.168.202.129@tcp [ 2368.945375] Key type .llcrypt unregistered [ 2368.946910] Key type ._llcrypt unregistered [ 2378.654807] Key type ._llcrypt registered [ 2378.656158] Key type .llcrypt registered [ 2378.697708] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 2379.192302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2379.207482] alg: No test for adler32 (adler32-zlib) [ 2380.075661] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 2380.168580] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 2381.752177] Key type lgssc registered [ 2382.161273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2385.100362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2385.109131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.310078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2387.845470] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2391.244631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2391.430384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2393.091336] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2395.164777] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2396.597104] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2399.259206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2399.392663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2401.478593] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2403.604838] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2410.663634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 2417.027622] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2418.627784] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2418.630694] Lustre: Skipped 2 previous similar messages [ 2420.614666] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2420.618083] Lustre: Skipped 2 previous similar messages [ 2426.899177] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2426.910418] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2430.098940] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.29@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2430.104206] Lustre: Skipped 2 previous similar messages [ 2432.055579] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.29@tcp 192.168.0.[2,10]@tcp [ 2438.416690] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.29/24@tcp in log lustre-MDT0000 [ 2438.420442] Lustre: Skipped 2 previous similar messages [ 2438.459300] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.29/24@tcp [ 2438.462079] Lustre: Skipped 1 previous similar message [ 2444.791801] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.29/24@tcp [ 2451.937722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2451.939301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.940453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2451.943577] Lustre: Skipped 1 previous similar message [ 2454.126457] LustreError: 129683:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2454.165930] Lustre: server umount lustre-OST0000 complete [ 2457.057615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.058155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2457.060821] Lustre: Skipped 1 previous similar message [ 2457.062368] Lustre: Skipped 1 previous similar message [ 2461.631370] LustreError: 129884:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2461.634029] LustreError: 129884:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2461.657245] Lustre: server umount lustre-MDT0000 complete [ 2462.176876] LustreError: 126061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2462.181055] LustreError: 126061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2462.930082] LustreError: 126085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353643 with bad export cookie 17367147542316282468 [ 2462.932844] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2462.934025] LustreError: 126085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2462.997983] LustreError: 130086:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2463.000725] LustreError: 130086:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2463.056373] Lustre: server umount lustre-MDT0001 complete [ 2466.623164] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 2467.684590] Key type lgssc unregistered [ 2467.819487] LNet: 130617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2467.822568] LNetError: 130617:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2467.830341] LNet: Removed LNI 192.168.202.129@tcp [ 2468.146725] Key type .llcrypt unregistered [ 2468.147900] Key type ._llcrypt unregistered [ 2478.664349] Key type ._llcrypt registered [ 2478.665718] Key type .llcrypt registered [ 2478.704872] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 2479.079613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2479.113813] alg: No test for adler32 (adler32-zlib) [ 2479.981222] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 2480.067308] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 2481.656142] Key type lgssc registered [ 2482.065761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2484.840546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2484.848331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2486.000171] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2486.004099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2486.009453] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.29/24@tcp [ 2487.299174] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2490.153152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2490.264468] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2490.273891] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.29/24@tcp [ 2490.288580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2491.689510] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2493.451554] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2494.662355] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2497.047690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2497.163154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2499.081837] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2500.877698] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2511.535945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 2515.937726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2515.939663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2515.943311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2516.962373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2516.965254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2516.965509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2520.687113] LustreError: 134457:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2520.728571] Lustre: server umount lustre-OST0000 complete [ 2522.082049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2522.082778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2522.086190] Lustre: Skipped 1 previous similar message [ 2522.089300] Lustre: Skipped 2 previous similar messages [ 2527.201575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2527.204296] Lustre: Skipped 1 previous similar message [ 2528.130634] LustreError: 134659:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2528.132275] LustreError: 134659:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2528.157071] Lustre: server umount lustre-MDT0000 complete [ 2529.507167] LustreError: 133757:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353709 with bad export cookie 1621350964592551533 [ 2529.509765] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2529.512118] LustreError: 133757:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2529.574174] LustreError: 134861:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2529.576340] LustreError: 134861:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2529.631580] Lustre: server umount lustre-MDT0001 complete [ 2533.195337] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 2534.271695] Key type lgssc unregistered [ 2534.402466] LNet: 135393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2534.405536] LNetError: 135393:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2534.412383] LNet: Removed LNI 192.168.202.129@tcp [ 2534.714754] Key type .llcrypt unregistered [ 2534.715963] Key type ._llcrypt unregistered [ 2544.193803] Key type ._llcrypt registered [ 2544.195440] Key type .llcrypt registered [ 2544.248453] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 2551.012319] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 2551.375740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2551.420073] alg: No test for adler32 (adler32-zlib) [ 2552.285654] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 2552.376514] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 2553.968219] Key type lgssc registered [ 2554.358703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2557.537200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2559.886229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2562.078256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2564.260680] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2566.532553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2566.551179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2566.555971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.636374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2567.648457] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2567.685204] Lustre: lustre-MDT0000: new disk, initializing [ 2567.710776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2567.717327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2569.027669] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2573.199531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2573.218646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2573.251399] Lustre: 138739:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2573.277374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2573.279652] Lustre: Skipped 1 previous similar message [ 2573.316564] Lustre: lustre-MDT0001: new disk, initializing [ 2573.348402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2573.360052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2573.363107] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2574.742258] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2577.899703] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2579.100190] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2581.497556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2581.525335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2581.623649] Lustre: lustre-OST0000: new disk, initializing [ 2581.625976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2581.654303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2583.671475] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2586.921476] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2588.359300] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2590.711281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2590.714679] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2590.747532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2591.488524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2592.914586] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2592.995057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2595.810219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2595.810730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2595.812407] LustreError: Skipped 1 previous similar message [ 2595.817740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2595.820686] Lustre: Skipped 1 previous similar message [ 2599.789109] LustreError: 141702:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2599.831796] Lustre: server umount lustre-OST0000 complete [ 2604.000938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2604.003231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.006356] Lustre: Skipped 1 previous similar message [ 2604.007984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2606.049989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2606.050064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2606.058042] Lustre: Skipped 1 previous similar message [ 2607.300134] LustreError: 141903:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2607.302060] LustreError: 141903:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2607.332429] Lustre: server umount lustre-MDT0000 complete [ 2608.682046] LustreError: 140490:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353789 with bad export cookie 4052642550844618909 [ 2608.685177] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2608.685335] LustreError: 140490:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2608.748089] LustreError: 142104:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2608.750298] LustreError: 142104:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2608.804696] Lustre: server umount lustre-MDT0001 complete [ 2611.865813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2612.062681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2613.529828] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2616.906284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2617.032396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2618.501922] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2620.510623] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2621.880937] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2624.370041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2626.614972] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2628.507305] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2629.930711] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2637.147551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 2638.420306] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2638.488274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2641.915800] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2641.944642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2642.008279] Lustre: lustre-OST0001: new disk, initializing [ 2642.010105] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2642.042525] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2642.045553] Lustre: Skipped 1 previous similar message [ 2643.900727] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2647.220953] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2650.627951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2650.632337] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2650.662507] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2651.376040] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2652.744671] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2652.814744] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2654.045767] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cced0517000.ost_server_uuid 50 [ 2654.586509] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8cced0517000.ost_server_uuid in FULL state after 0 sec [ 2660.836730] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2660.836781] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.840358] LustreError: Skipped 1 previous similar message [ 2660.844551] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2660.847208] Lustre: Skipped 1 previous similar message [ 2662.572458] LustreError: 146908:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2662.574283] LustreError: 146908:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2662.597368] Lustre: server umount lustre-OST0001 complete [ 2665.954395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2665.955656] LustreError: 144642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2665.956979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2665.958573] Lustre: Skipped 2 previous similar messages [ 2665.967072] LustreError: 144642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2671.074662] LustreError: 144178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2671.079828] LustreError: 144178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2676.193541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2676.195778] Lustre: Skipped 3 previous similar messages [ 2676.197069] LustreError: 144177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.202131] LustreError: 144177:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2678.240121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2678.260576] LustreError: 147109:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2678.263161] LustreError: 147109:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2678.302641] Lustre: server umount lustre-OST0000 complete [ 2681.314069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2685.786705] Lustre: server umount lustre-MDT0000 complete [ 2686.433077] LustreError: 143292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.440933] LustreError: 143292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2687.023147] LustreError: 142561:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353867 with bad export cookie 4052642550844619777 [ 2687.026395] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2687.027299] LustreError: 142561:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2687.092647] LustreError: 147514:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2687.095143] LustreError: 147514:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2687.149621] Lustre: server umount lustre-MDT0001 complete [ 2693.362688] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 2694.379948] Key type lgssc unregistered [ 2694.514416] LNet: 148503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2694.519203] LNetError: 148503:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2694.529273] LNet: Removed LNI 192.168.202.129@tcp [ 2694.797226] Key type .llcrypt unregistered [ 2694.798164] Key type ._llcrypt unregistered [ 2705.356711] Key type ._llcrypt registered [ 2705.357966] Key type .llcrypt registered [ 2705.393809] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 2711.290136] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 2711.690731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2711.703524] alg: No test for adler32 (adler32-zlib) [ 2712.554163] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 2712.639515] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 2714.224109] Key type lgssc registered [ 2714.565276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2717.443247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2719.551218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2721.522854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2723.633169] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2725.927846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2725.949707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2725.955859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2727.043123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2727.055078] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2727.095170] Lustre: lustre-MDT0000: new disk, initializing [ 2727.125982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.134427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2728.486396] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2732.617614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2732.640428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2732.673766] Lustre: 152424:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2732.697641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2732.699790] Lustre: Skipped 1 previous similar message [ 2732.734989] Lustre: lustre-MDT0001: new disk, initializing [ 2732.766477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2732.779391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2732.783386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2734.037078] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2737.123569] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2738.340676] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2740.556021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2740.584041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2740.676893] Lustre: lustre-OST0000: new disk, initializing [ 2740.678711] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2740.705147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2741.111975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2741.115898] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2741.144947] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2742.561683] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2745.752547] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2747.235219] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2747.314987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2748.627859] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2748.702901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2751.458034] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2751.460274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2751.460349] LustreError: Skipped 1 previous similar message [ 2751.464085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2751.466970] Lustre: Skipped 1 previous similar message [ 2755.437088] LustreError: 155363:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2755.474383] Lustre: server umount lustre-OST0000 complete [ 2758.625068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2758.628164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.633038] Lustre: Skipped 1 previous similar message [ 2758.635345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2761.697588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2761.697861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.814388] LustreError: 155564:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2762.816016] LustreError: 155564:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2762.843160] Lustre: server umount lustre-MDT0000 complete [ 2764.120769] LustreError: 152416:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353944 with bad export cookie 6460056326372798983 [ 2764.125536] LustreError: 152416:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2764.125623] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2764.191461] LustreError: 155766:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2764.194242] LustreError: 155766:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2764.249591] Lustre: server umount lustre-MDT0001 complete [ 2767.016288] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 23:32:25 (1763353945) [ 2772.166564] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 2775.067964] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 2778.144202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2780.083922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2781.878944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2783.930264] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2786.273738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2786.296647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.381216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2786.392900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2786.428956] Lustre: lustre-MDT0000: new disk, initializing [ 2786.453560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.460370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2787.760250] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2792.130272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2792.160329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2792.205256] Lustre: 158751:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2792.235509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2792.238231] Lustre: Skipped 1 previous similar message [ 2792.278881] Lustre: lustre-MDT0001: new disk, initializing [ 2792.316536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2792.330906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2792.337891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2793.662888] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2796.745182] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2797.931928] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2800.191306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2800.215091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2800.305971] Lustre: lustre-OST0000: new disk, initializing [ 2800.307787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2800.336236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.245284] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2804.734970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2804.738722] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2804.774277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2805.434248] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2807.463451] Lustre: Failing over lustre-OST0000 [ 2807.488103] LustreError: 161194:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2807.490663] LustreError: 161194:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2807.530749] Lustre: server umount lustre-OST0000 complete [ 2820.391422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2820.510481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.517974] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2821.858870] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2822.422679] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2823.688405] Lustre: Failing over lustre-MDT0000 [ 2823.754072] LustreError: 162166:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2823.755577] LustreError: 162166:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2823.780204] Lustre: server umount lustre-MDT0000 complete [ 2825.697363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.702108] Lustre: Skipped 3 previous similar messages [ 2825.737148] Lustre: 150642:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763353990/real 1763353990] req@000000000a5e08fa x1849010571762816/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763354006 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2830.816191] Lustre: 150640:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763353995/real 1763353995] req@00000000995010d8 x1849010571764608/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763354011 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2830.816933] LustreError: 160454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.829849] LustreError: 160454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2832.089759] LustreError: 159484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.936219] Lustre: 150641:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763354000/real 1763354000] req@0000000061768c12 x1849010571765760/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763354016 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2835.937447] LustreError: 160454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.950420] LustreError: 160454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2836.652940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2836.687460] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2836.691644] Lustre: 158753:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.202.129@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2836.760659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2836.800409] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2837.209503] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2838.025515] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2842.087402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2842.096472] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2842.100719] Lustre: Skipped 1 previous similar message [ 2842.115804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2847.202966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2847.203203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2847.205132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2847.205137] Lustre: Skipped 1 previous similar message [ 2847.205294] LustreError: Skipped 1 previous similar message [ 2847.208265] Lustre: Skipped 1 previous similar message [ 2849.133126] LustreError: 163048:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2849.135191] LustreError: 163048:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2849.175802] Lustre: server umount lustre-OST0000 complete [ 2852.322152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2852.322512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2852.328151] Lustre: Skipped 2 previous similar messages [ 2856.540301] Lustre: server umount lustre-MDT0000 complete [ 2857.441049] LustreError: 160454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.446705] LustreError: 160454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2857.654188] LustreError: 158741:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354038 with bad export cookie 6460056326372801041 [ 2857.656851] LustreError: 158741:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2857.658024] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2857.762390] Lustre: server umount lustre-MDT0001 complete [ 2861.139595] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 2862.130963] Key type lgssc unregistered [ 2862.253331] LNet: 163984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2862.255570] LNetError: 163984:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2862.264286] LNet: Removed LNI 192.168.202.129@tcp [ 2862.556074] Key type .llcrypt unregistered [ 2862.557068] Key type ._llcrypt unregistered [ 2881.960407] Key type ._llcrypt registered [ 2881.961299] Key type .llcrypt registered [ 2881.995418] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 2882.342634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2882.393636] alg: No test for adler32 (adler32-zlib) [ 2883.243027] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 2883.320546] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 2884.904120] Key type lgssc registered [ 2885.270128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2887.880207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2887.885924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2889.055435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2890.427229] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2893.363404] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2895.330138] LustreError: 165511:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2895.333815] LustreError: 165511:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2895.336724] LustreError: 165511:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2895.400792] LustreError: 166032:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2895.460888] Lustre: server umount lustre-MDT0000 complete [ 2900.605472] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 2903.700957] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 2907.254495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2909.594932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2911.748710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2913.820484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2916.196776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2916.222759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2916.303631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2916.314610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2916.348848] Lustre: lustre-MDT0000: new disk, initializing [ 2916.373376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2916.379895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2917.635949] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2921.664799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2921.688489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2921.722877] Lustre: 168671:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2921.746196] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2921.747992] Lustre: Skipped 1 previous similar message [ 2921.782921] Lustre: lustre-MDT0001: new disk, initializing [ 2921.815117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2921.826478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2921.829298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2923.061668] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2926.022940] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2927.109958] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2929.246294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2929.269766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2929.358375] Lustre: lustre-OST0000: new disk, initializing [ 2929.359878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2929.385654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2929.772562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2929.775474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2929.811828] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2931.215766] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 2934.219528] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2936.001591] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2936.560084] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 23:35:15 (1763354115) [ 3001.824553] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3001.824714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3001.828477] Lustre: Skipped 2 previous similar messages [ 3001.831113] Lustre: Skipped 2 previous similar messages [ 3006.944636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3006.946913] Lustre: Skipped 2 previous similar messages [ 3007.801934] LustreError: 172179:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3007.803526] LustreError: 172179:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3007.841939] Lustre: server umount lustre-MDT0000 complete [ 3008.974375] LustreError: 172080:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354189 with bad export cookie 10552587541292908751 [ 3008.975295] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3008.977568] LustreError: 172080:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3012.064355] LustreError: 171736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3012.064407] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3012.069943] LustreError: 171736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3012.075340] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3013.088366] LustreError: 171736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3018.208346] LustreError: 171736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3018.208391] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3018.215921] Lustre: Skipped 1 previous similar message [ 3023.328092] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3023.356500] LustreError: 172382:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3023.358167] LustreError: 172382:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3023.397246] Lustre: server umount lustre-MDT0001 complete [ 3030.702855] LustreError: 172585:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3030.705325] LustreError: 172585:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3030.737357] Lustre: server umount lustre-OST0000 complete [ 3033.790957] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 3036.933941] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 3040.137265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3042.113938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3043.979882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3045.949748] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3048.183802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3048.205466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.292705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3048.304598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3048.346219] Lustre: lustre-MDT0000: new disk, initializing [ 3048.374794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3048.382426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3049.647907] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3053.560655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3053.584985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3053.619764] Lustre: 174919:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3053.644608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3053.647467] Lustre: Skipped 1 previous similar message [ 3053.688088] Lustre: lustre-MDT0001: new disk, initializing [ 3053.733247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3053.736565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3055.012391] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3058.003527] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3059.120214] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3061.299635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3061.324367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3061.445786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3061.447900] Lustre: Skipped 1 previous similar message [ 3063.283695] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3066.229497] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3066.870626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3066.874107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3066.904039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3067.457800] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3067.521521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3068.674892] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3068.736488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3071.969784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3071.972639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3071.973811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3071.977169] Lustre: Skipped 1 previous similar message [ 3071.980377] Lustre: Skipped 1 previous similar message [ 3075.372122] LustreError: 177858:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3075.373983] LustreError: 177858:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3075.409030] Lustre: server umount lustre-OST0000 complete [ 3077.089747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.093128] Lustre: Skipped 1 previous similar message [ 3082.209101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3082.211797] Lustre: Skipped 4 previous similar messages [ 3082.905412] Lustre: server umount lustre-MDT0000 complete [ 3084.201293] LustreError: 174910:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354264 with bad export cookie 10552587541293067567 [ 3084.203940] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3084.205222] LustreError: 174910:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3084.275851] LustreError: 178261:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3084.278200] LustreError: 178261:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3084.336182] Lustre: server umount lustre-MDT0001 complete [ 3087.341824] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:37:46 (1763354266) [ 3092.232536] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 3094.981257] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 3097.963240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3099.858244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3101.660115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3103.634661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3105.892780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3105.916087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3106.013984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3106.027428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3106.030203] Lustre: Skipped 1 previous similar message [ 3106.083661] Lustre: lustre-MDT0000: new disk, initializing [ 3106.085254] Lustre: Skipped 1 previous similar message [ 3106.115970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3106.122767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3107.500875] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3111.681730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3111.709959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3111.748469] Lustre: 181245:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3111.852294] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3113.237092] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3116.383762] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3117.720719] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3119.972126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3119.997053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3121.937148] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3125.078547] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3128.825986] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3128.863182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3133.923299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3133.923688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.925816] LustreError: Skipped 2 previous similar messages [ 3133.926281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3133.930358] Lustre: Skipped 1 previous similar message [ 3138.415115] LustreError: 183792:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3138.417176] LustreError: 183792:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3138.457629] Lustre: server umount lustre-OST0000 complete [ 3142.624819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3142.627125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.400094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3154.460926] Lustre: server umount lustre-MDT0000 complete [ 3155.682244] LustreError: 181236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354336 with bad export cookie 10552587541293068435 [ 3155.685823] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3155.687642] LustreError: 181236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3155.806889] Lustre: server umount lustre-MDT0001 complete [ 3158.580765] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:38:57 (1763354337) [ 3163.787903] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 3166.935169] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 3170.029291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3172.084242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3174.066542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3176.238202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3178.554056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3178.577813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3178.662396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3178.674667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3178.676719] Lustre: Skipped 3 previous similar messages [ 3178.714919] Lustre: lustre-MDT0000: new disk, initializing [ 3178.716175] Lustre: Skipped 2 previous similar messages [ 3178.739438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3178.741221] Lustre: Skipped 2 previous similar messages [ 3178.746750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3178.749808] Lustre: Skipped 2 previous similar messages [ 3180.045831] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3184.192192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3184.214089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3184.254517] Lustre: 187180:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3184.354028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3185.688747] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3188.824946] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3190.021118] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3192.417144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3192.445894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3194.516433] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3197.644740] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3201.561105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3206.626946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.627444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3206.628142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3206.628146] Lustre: Skipped 8 previous similar messages [ 3206.630775] Lustre: Skipped 2 previous similar messages [ 3210.671070] LustreError: 189731:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3210.673817] LustreError: 189731:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3210.712284] Lustre: server umount lustre-OST0000 complete [ 3215.329047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3215.331465] LustreError: Skipped 1 previous similar message [ 3218.149934] Lustre: server umount lustre-MDT0000 complete [ 3219.419711] LustreError: 187173:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354399 with bad export cookie 10552587541293069506 [ 3219.424063] LustreError: 187173:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3219.424565] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3219.550377] Lustre: server umount lustre-MDT0001 complete [ 3223.150217] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 3224.214387] Key type lgssc unregistered [ 3224.341404] LNet: 190665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3224.345388] LNetError: 190665:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3224.354300] LNet: Removed LNI 192.168.202.129@tcp [ 3224.644197] Key type .llcrypt unregistered [ 3224.645429] Key type ._llcrypt unregistered [ 3233.519791] Key type ._llcrypt registered [ 3233.520823] Key type .llcrypt registered [ 3233.564459] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 3234.003579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3234.012078] alg: No test for adler32 (adler32-zlib) [ 3234.857705] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 3234.937118] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 3236.520093] Key type lgssc registered [ 3236.877063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3239.425317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3239.431477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3240.596516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3241.845550] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3244.612447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3244.745673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3246.060446] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3247.828179] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3249.049460] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3251.289368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3251.403274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3253.252071] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3255.142768] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3264.744326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3280.357596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.359723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3280.359765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3280.361526] Lustre: Skipped 1 previous similar message [ 3283.375137] LustreError: 194309:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3283.414284] Lustre: server umount lustre-OST0000 complete [ 3285.474055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3285.474784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3285.479815] Lustre: Skipped 1 previous similar message [ 3285.481608] Lustre: Skipped 1 previous similar message [ 3290.593158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3290.596801] Lustre: Skipped 2 previous similar messages [ 3290.751104] LustreError: 194511:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3290.754872] LustreError: 194511:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3290.780059] Lustre: server umount lustre-MDT0000 complete [ 3292.026550] LustreError: 194210:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354472 with bad export cookie 7554383108217021747 [ 3292.029910] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3292.031810] LustreError: 194210:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3292.104806] LustreError: 194713:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3292.106580] LustreError: 194713:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3292.155350] Lustre: server umount lustre-MDT0001 complete [ 3295.779848] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 3296.768115] Key type lgssc unregistered [ 3296.893426] LNet: 195244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3296.895788] LNetError: 195244:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3296.901303] LNet: Removed LNI 192.168.202.129@tcp [ 3297.202769] Key type .llcrypt unregistered [ 3297.203752] Key type ._llcrypt unregistered [ 3306.312285] Key type ._llcrypt registered [ 3306.313240] Key type .llcrypt registered [ 3306.349228] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 3306.691483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3306.738044] alg: No test for adler32 (adler32-zlib) [ 3307.593623] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 3307.683188] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 3309.288147] Key type lgssc registered [ 3309.686674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3312.367545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3312.373149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3313.521398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.768322] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3317.509170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3317.623399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3318.899286] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3320.730179] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3321.896895] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3324.221978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3324.352208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3326.282394] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3328.042269] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3337.704413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3342.819431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.820612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3342.821289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3342.823077] Lustre: Skipped 1 previous similar message [ 3342.826749] LustreError: Skipped 1 previous similar message [ 3343.587281] Lustre: lustre-OST0000: Not available for connect from 192.168.202.29@tcp (stopping) [ 3343.591793] Lustre: Skipped 1 previous similar message [ 3346.735103] LustreError: 198884:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3346.774687] Lustre: server umount lustre-OST0000 complete [ 3348.635275] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8ccec773c800.ost_server_uuid 50 [ 3349.118666] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8ccec773c800.ost_server_uuid in DISCONN state after 0 sec [ 3349.680361] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3362.723085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3362.723962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3362.727109] Lustre: Skipped 1 previous similar message [ 3365.824140] LustreError: 199280:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3365.827520] LustreError: 199280:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3365.863289] Lustre: server umount lustre-MDT0000 complete [ 3367.139543] LustreError: 196600:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354547 with bad export cookie 15441551911961653756 [ 3367.144700] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3367.218672] LustreError: 199483:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3367.222031] LustreError: 199483:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3367.282409] Lustre: server umount lustre-MDT0001 complete [ 3370.115354] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:42:28 (1763354548) [ 3372.373898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3372.539295] LustreError: 200153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3372.544509] LustreError: 200153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3372.566085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3373.939708] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3376.757368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3376.892814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3378.297526] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3380.232878] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3381.409206] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3383.734730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3385.679452] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3387.425281] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3389.667395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3389.695880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3389.753786] Lustre: lustre-OST0001: new disk, initializing [ 3389.756160] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3389.789141] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3389.792713] Lustre: Skipped 1 previous similar message [ 3391.689349] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3394.785758] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3396.095258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3396.098235] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3396.131850] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3397.161409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3402.214657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3402.215348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3402.227029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3402.230427] Lustre: Skipped 2 previous similar messages [ 3407.222129] LustreError: 203360:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3407.226817] LustreError: 203360:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3407.253557] Lustre: server umount lustre-OST0000 complete [ 3407.335531] LustreError: 203265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3407.344461] LustreError: 203265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3409.093290] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3409.168107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3410.451101] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3410.521612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3410.651568] LustreError: 201749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3415.770717] LustreError: 203271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3415.775199] LustreError: 203271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3421.666339] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3421.666367] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3421.671790] LustreError: Skipped 2 previous similar messages [ 3421.679653] Lustre: Skipped 1 previous similar message [ 3421.683854] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3421.689249] Lustre: Skipped 2 previous similar messages [ 3422.690201] LustreError: 203265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3422.695413] LustreError: 203265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3426.541083] LustreError: 204064:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3426.542716] LustreError: 204064:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3426.575758] Lustre: server umount lustre-OST0001 complete [ 3428.320866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3428.323495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3432.929752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3432.932041] Lustre: Skipped 4 previous similar messages [ 3442.144204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3442.187103] LustreError: 204265:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3442.189362] LustreError: 204265:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3442.220346] Lustre: server umount lustre-MDT0000 complete [ 3443.170049] LustreError: 200152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3443.174219] LustreError: 200152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3443.494420] LustreError: 200134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354623 with bad export cookie 15441551911961654813 [ 3443.498272] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3443.499875] LustreError: 200134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3443.633995] Lustre: server umount lustre-MDT0001 complete [ 3445.511562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3446.812561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3448.063340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3449.363889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3451.592413] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:43:50 (1763354630) [ 3453.830427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.922872] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3453.935698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3454.018654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3455.321302] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3458.114793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3458.149754] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3458.164054] Lustre: 205958:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3459.518605] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3461.456540] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3462.637737] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3464.955332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3465.020251] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3466.919130] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3468.682717] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3470.939774] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3470.962757] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3472.043757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3472.849934] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3474.599980] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3477.475890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3477.476895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3477.479081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3477.479085] Lustre: Skipped 2 previous similar messages [ 3477.479603] Lustre: Skipped 2 previous similar messages [ 3477.481517] LustreError: Skipped 1 previous similar message [ 3481.517115] LustreError: 209148:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3481.519177] LustreError: 209148:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3481.535894] Lustre: server umount lustre-OST0000 complete [ 3482.597422] LustreError: 207586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3482.602183] LustreError: 207586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3492.835635] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3492.837860] LustreError: Skipped 1 previous similar message [ 3498.257870] Lustre: server umount lustre-OST0001 complete [ 3499.491873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3499.495619] Lustre: Skipped 2 previous similar messages [ 3505.625706] Lustre: server umount lustre-MDT0000 complete [ 3506.832084] LustreError: 205949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354687 with bad export cookie 15441551911961656199 [ 3506.835636] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3506.838608] LustreError: 205949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3506.953405] Lustre: server umount lustre-MDT0001 complete [ 3508.776691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3510.029972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3511.262023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3512.525511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3514.795695] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:44:53 (1763354693) [ 3520.174261] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 3523.062826] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 3526.284368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3528.832070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3531.567735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3533.734711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3536.253496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3536.277202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3536.353348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3536.363162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3536.399820] Lustre: lustre-MDT0000: new disk, initializing [ 3536.423473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3536.427283] Lustre: Skipped 3 previous similar messages [ 3536.433758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3537.740692] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3542.300042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3542.330348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3542.373312] Lustre: 213568:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3542.398859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3542.400890] Lustre: Skipped 1 previous similar message [ 3542.440044] Lustre: lustre-MDT0001: new disk, initializing [ 3542.485150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3542.488709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3543.978286] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3547.543237] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3548.709250] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3550.847595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3550.874195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3550.963044] Lustre: lustre-OST0000: new disk, initializing [ 3550.964715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3552.821064] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3555.878042] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3556.346212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3556.350524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3556.378239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3557.196441] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3557.263557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3558.506422] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3558.572836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3561.446156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3561.446858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3561.449130] LustreError: Skipped 1 previous similar message [ 3561.452513] Lustre: Skipped 1 previous similar message [ 3561.453479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3561.458239] Lustre: Skipped 10 previous similar messages [ 3565.293113] LustreError: 216506:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3565.295784] LustreError: 216506:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3565.336207] Lustre: server umount lustre-OST0000 complete [ 3572.706961] Lustre: server umount lustre-MDT0000 complete [ 3573.946079] LustreError: 213561:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354754 with bad export cookie 15441551911961657648 [ 3573.949216] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3573.950518] LustreError: 213561:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3574.083339] Lustre: server umount lustre-MDT0001 complete [ 3576.855668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3577.010783] LustreError: 217385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3577.017105] LustreError: 217385:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3578.304553] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3581.198629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3582.597273] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3584.349363] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3585.588869] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3587.902312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3589.837307] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3591.579594] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3593.009323] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3603.267762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3604.527265] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3604.603549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3606.497036] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3611.352107] Lustre: server umount lustre-OST0000 complete [ 3613.206859] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3614.497650] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3638.005023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3638.138350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3638.141086] Lustre: Skipped 5 previous similar messages [ 3640.019084] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3641.804864] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3642.361646] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3645.356070] LustreError: 221829:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3645.357950] LustreError: 221829:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3645.389195] Lustre: server umount lustre-OST0000 complete [ 3646.582070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3646.582611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3646.587100] Lustre: Skipped 6 previous similar messages [ 3646.588689] Lustre: Skipped 7 previous similar messages [ 3648.729261] Lustre: server umount lustre-MDT0000 complete [ 3649.988747] LustreError: 217410:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354830 with bad export cookie 15441551911961658530 [ 3649.992934] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3649.995075] LustreError: 217410:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3650.136662] Lustre: server umount lustre-MDT0001 complete [ 3652.838328] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:47:11 (1763354831) [ 3654.978395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3655.121668] LustreError: 222905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3656.360903] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3659.043318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3660.573184] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3662.270368] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3663.417872] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3665.598360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3667.595558] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3669.301105] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3670.577848] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3680.827830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3682.085188] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3682.156550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3684.322257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3684.345784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3684.401991] Lustre: lustre-OST0001: new disk, initializing [ 3684.404028] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3686.298441] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3689.305765] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3690.609256] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3693.054238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3693.058894] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3693.089666] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3693.762502] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3694.995239] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3695.061068] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3698.147411] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3698.149648] LustreError: Skipped 1 previous similar message [ 3701.760102] Lustre: server umount lustre-OST0001 complete [ 3703.707905] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3703.783631] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3705.162926] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3705.260857] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3728.844417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3730.884449] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3732.704976] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3733.251795] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3739.959371] Lustre: server umount lustre-OST0001 complete [ 3766.240150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3766.293441] Lustre: server umount lustre-OST0000 complete [ 3773.758404] LustreError: 229277:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3773.760318] LustreError: 229277:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 3773.781406] Lustre: server umount lustre-MDT0000 complete [ 3774.966137] LustreError: 222884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354955 with bad export cookie 15441551911961659461 [ 3774.969463] LustreError: 222884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3774.969781] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3789.280134] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3789.366813] Lustre: server umount lustre-MDT0001 complete [ 3791.190377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3792.465290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3793.668579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3794.940706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3797.096116] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:49:35 (1763354975) [ 3799.286532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3799.385792] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3799.398566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3799.483302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3799.486446] Lustre: Skipped 5 previous similar messages [ 3800.804729] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3803.526970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3803.559603] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3803.571232] Lustre: 230972:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3804.899714] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3806.655327] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3807.914778] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3810.188390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3810.251905] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3812.181256] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3813.982199] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3822.751156] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3822.773680] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3824.767068] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3826.549098] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3828.089239] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 3829.401136] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3829.471935] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3830.724176] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3830.795229] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3831.915571] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cced02d4000.ost_server_uuid 50 [ 3832.381930] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8cced02d4000.ost_server_uuid in FULL state after 0 sec [ 3832.692299] Lustre: Permanently deactivating lustre-OST0001 [ 3832.696651] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3833.199218] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3833.445692] Lustre: Permanently reactivating lustre-OST0001 [ 3840.509347] Lustre: server umount lustre-OST0001 complete [ 3843.041361] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3843.041843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.043887] LustreError: Skipped 2 previous similar messages [ 3843.044717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3843.046591] Lustre: Skipped 7 previous similar messages [ 3843.049898] LustreError: 232595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3843.050640] Lustre: Skipped 19 previous similar messages [ 3843.054149] LustreError: 232595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 30 previous similar messages [ 3847.820572] Lustre: server umount lustre-OST0000 complete [ 3859.447606] Lustre: server umount lustre-MDT0000 complete [ 3860.612879] LustreError: 231009:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355041 with bad export cookie 15441551911961660721 [ 3860.615934] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3863.056095] LustreError: 235167:0:(osp_dev.c:692:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 3863.059970] LustreError: 230974:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3863.176259] Lustre: server umount lustre-MDT0001 complete [ 3865.054716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3866.319925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3867.539664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3868.783437] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3870.883733] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:50:49 (1763355049) [ 3871.197204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3873.328714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3873.415823] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3873.425346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3873.427968] Lustre: Skipped 5 previous similar messages [ 3874.748550] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3877.471140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3877.500326] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3877.508692] Lustre: 237116:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3878.820704] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3880.518483] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3881.712307] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3883.922053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3883.950860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3884.021523] Lustre: Permanently deactivating lustre-OST0000 [ 3884.027034] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3885.926962] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3887.649971] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3887.983684] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3887.985653] Lustre: Skipped 1 previous similar message [ 3889.910335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3891.782854] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3893.504259] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3896.343202] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3899.501614] Lustre: Permanently reactivating lustre-OST0000 [ 3905.528816] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3905.531523] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3905.535706] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3905.551577] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3905.557947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3912.448566] Lustre: server umount lustre-OST0001 complete [ 3919.824224] Lustre: server umount lustre-OST0000 complete [ 3927.193946] Lustre: server umount lustre-MDT0000 complete [ 3928.390219] LustreError: 237107:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355108 with bad export cookie 15441551911961662422 [ 3928.393604] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3928.395081] LustreError: 237107:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3928.522048] Lustre: server umount lustre-MDT0001 complete [ 3931.950598] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 3932.945399] Key type lgssc unregistered [ 3933.083325] LNet: 241510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3933.085395] LNetError: 241510:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3933.092303] LNet: Removed LNI 192.168.202.129@tcp [ 3933.406885] Key type .llcrypt unregistered [ 3933.408553] Key type ._llcrypt unregistered [ 3941.788237] Key type ._llcrypt registered [ 3941.789187] Key type .llcrypt registered [ 3941.824885] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 3942.127667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3942.161519] alg: No test for adler32 (adler32-zlib) [ 3943.001088] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 3943.080361] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 3944.656161] Key type lgssc registered [ 3945.009505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3946.119810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3948.289168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3948.294697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3949.463978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3950.753873] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3953.514265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3953.537393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3953.568405] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3953.577526] Lustre: 242926:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3953.581173] Lustre: Permanently deactivating lustre-MDT0001 [ 3953.585949] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3953.694860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3954.972580] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3956.674929] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3957.836478] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3958.256109] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3958.259852] LustreError: 242967:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3958.263660] LustreError: 242967:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3958.267404] LustreError: 242967:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 3960.085192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3960.197716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3962.087940] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3963.813205] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3964.324359] LustreError: 244556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3964.331266] LustreError: 244556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3966.064328] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3966.133603] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3966.309283] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3967.993599] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 3969.717947] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3974.503956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3974.505227] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3975.299550] Lustre: Permanently reactivating lustre-MDT0001 [ 3975.306457] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3975.308840] Lustre: Skipped 2 previous similar messages [ 3976.096514] Lustre: lustre-MDT0001: Client 1283c1b0-9b89-4986-96bc-b0b113bbba8b (at 192.168.202.29@tcp) reconnecting [ 3977.518529] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3978.048583] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3979.614690] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3979.681705] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3980.960908] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3982.056730] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 3982.829090] Lustre: Permanently deactivating lustre-MDT0001 [ 3982.835340] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3982.838867] Lustre: Skipped 2 previous similar messages [ 3988.480207] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3994.592932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3994.595175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3994.598775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3997.155965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3997.156186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3997.158440] Lustre: Skipped 2 previous similar messages [ 3997.162901] Lustre: Skipped 2 previous similar messages [ 3998.783304] LustreError: 247055:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3998.811372] Lustre: server umount lustre-MDT0000 complete [ 4000.019951] LustreError: 242917:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355180 with bad export cookie 15295512992017608936 [ 4000.023877] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4000.026035] LustreError: 242917:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4000.095705] LustreError: 247258:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4000.100235] LustreError: 247258:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4000.147622] Lustre: server umount lustre-MDT0001 complete [ 4007.471193] LustreError: 247458:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4007.473212] LustreError: 247458:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4007.482633] Lustre: server umount lustre-OST0000 complete [ 4011.093082] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:53:09 (1763355189) [ 4014.524595] LustreError: 248463:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4014.526607] LustreError: 248463:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4014.576519] Lustre: server umount lustre-OST0001 complete [ 4016.666974] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 4019.849681] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 4023.170845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4025.267360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4027.366810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4029.540377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4031.741547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4031.765403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4031.842977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4031.846266] Lustre: Skipped 2 previous similar messages [ 4031.854630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4031.893268] Lustre: lustre-MDT0000: new disk, initializing [ 4031.915739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4031.922498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4033.182828] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4037.162399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4037.187455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4037.219036] Lustre: 250642:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4037.241514] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4037.242867] Lustre: Skipped 1 previous similar message [ 4037.281767] Lustre: lustre-MDT0001: new disk, initializing [ 4037.322616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4037.326538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4038.596313] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4041.624550] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4042.794218] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4044.965767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4045.005517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4045.104152] Lustre: lustre-OST0000: new disk, initializing [ 4045.107199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4045.139320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4045.141478] Lustre: Skipped 1 previous similar message [ 4046.995308] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4048.440933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4048.443650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4048.474900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4049.980537] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4050.671514] LustreError: 250648:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4054.862721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4054.887893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4054.944504] Lustre: lustre-OST0001: new disk, initializing [ 4054.946457] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4056.795475] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4058.303977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4058.307538] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4058.338228] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4059.872402] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4060.429225] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4060.760120] LustreError: 250648:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4060.768812] LustreError: 250648:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4070.864076] LustreError: 250648:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4073.954395] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4073.956558] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4073.959097] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4073.959618] Lustre: Skipped 1 previous similar message [ 4073.961699] Lustre: Skipped 1 previous similar message [ 4077.745482] LustreError: 254157:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4077.747725] LustreError: 254157:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4077.771240] Lustre: server umount lustre-OST0001 complete [ 4079.080855] LustreError: 252370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4079.086958] LustreError: 252370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4084.198989] LustreError: 252370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4084.203521] LustreError: 252370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4089.318566] LustreError: 252370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4089.323689] LustreError: 252370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4094.434272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4094.434462] LustreError: 252370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4094.436440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4094.436439] LustreError: Skipped 2 previous similar messages [ 4094.437290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4094.437295] Lustre: Skipped 1 previous similar message [ 4094.440973] LustreError: 252370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4094.443957] Lustre: Skipped 1 previous similar message [ 4099.554893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4099.556546] Lustre: Skipped 2 previous similar messages [ 4104.160148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4104.178531] LustreError: 254380:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4104.180371] LustreError: 254380:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4104.216689] Lustre: server umount lustre-OST0000 complete [ 4109.280749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4109.283059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.286672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4111.709386] Lustre: server umount lustre-MDT0000 complete [ 4112.975382] LustreError: 251422:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355293 with bad export cookie 15295512992017611505 [ 4112.978610] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4112.978974] LustreError: 251422:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4119.160941] Lustre: server umount lustre-MDT0001 complete [ 4122.648477] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 4123.727510] Key type lgssc unregistered [ 4123.846374] LNet: 255317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4123.848455] LNetError: 255317:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4123.856284] LNet: Removed LNI 192.168.202.129@tcp [ 4124.158294] Key type .llcrypt unregistered [ 4124.159327] Key type ._llcrypt unregistered [ 4135.634509] Key type ._llcrypt registered [ 4135.636148] Key type .llcrypt registered [ 4135.679443] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 4142.268506] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 4142.711387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4142.719702] alg: No test for adler32 (adler32-zlib) [ 4143.573167] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 4143.657433] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 4145.240117] Key type lgssc registered [ 4145.590185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4148.540431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4150.683813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4152.765236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4155.030301] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4157.330067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4157.350327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4157.354987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4158.437450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4158.449184] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4158.492307] Lustre: lustre-MDT0000: new disk, initializing [ 4158.519078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4158.525961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4159.792735] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4163.920276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4163.946676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4163.979774] Lustre: 259239:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4164.002243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4164.003942] Lustre: Skipped 1 previous similar message [ 4164.039825] Lustre: lustre-MDT0001: new disk, initializing [ 4164.074664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4164.089101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4164.092365] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4165.440115] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4168.604478] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4169.882185] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4172.172678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4172.196237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4172.288155] Lustre: lustre-OST0000: new disk, initializing [ 4172.290142] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4172.315454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4173.179564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4173.182829] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4173.216329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4174.136208] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4177.116337] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4178.459360] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4178.535424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4179.856934] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4179.927497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4183.523213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4183.523447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4183.525886] LustreError: Skipped 1 previous similar message [ 4183.526338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4183.531260] Lustre: Skipped 1 previous similar message [ 4183.538189] Lustre: Skipped 1 previous similar message [ 4186.605717] LustreError: 262172:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4186.639827] Lustre: server umount lustre-OST0000 complete [ 4188.642037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4188.642608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4188.646637] Lustre: Skipped 1 previous similar message [ 4188.649080] Lustre: Skipped 1 previous similar message [ 4193.761343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4193.763949] Lustre: Skipped 1 previous similar message [ 4193.984382] LustreError: 262374:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4193.986390] LustreError: 262374:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4194.012310] Lustre: server umount lustre-MDT0000 complete [ 4195.202178] LustreError: 262375:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355375 with bad export cookie 8257597065794002573 [ 4195.206752] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4195.207065] LustreError: 262375:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4195.279266] LustreError: 262576:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4195.282053] LustreError: 262576:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4195.340593] Lustre: server umount lustre-MDT0001 complete [ 4198.256236] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 23:56:17 (1763355377) [ 4200.406063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4200.585128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4201.914820] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4204.604863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4204.731207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4206.024290] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4207.758871] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4208.882923] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4211.068744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4213.042220] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4214.704987] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4271.588927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4271.590343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4271.590956] LustreError: Skipped 1 previous similar message [ 4271.597244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4271.601409] Lustre: Skipped 1 previous similar message [ 4274.543104] LustreError: 265590:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4274.544961] LustreError: 265590:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4274.590240] Lustre: server umount lustre-OST0000 complete [ 4275.986228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4280.072236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4280.181080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4280.184224] Lustre: Skipped 1 previous similar message [ 4282.048966] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4283.760861] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4292.451984] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4292.456041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4292.456110] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4292.462537] Lustre: Skipped 1 previous similar message [ 4318.180041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4318.180320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4318.182113] LustreError: Skipped 1 previous similar message [ 4318.182627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4318.188161] Lustre: Skipped 2 previous similar messages [ 4318.194183] Lustre: Skipped 1 previous similar message [ 4323.179271] LustreError: 267253:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4323.180976] LustreError: 267253:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4323.222754] Lustre: server umount lustre-OST0000 complete [ 4327.904805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4327.906913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4327.910365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4330.529474] Lustre: server umount lustre-MDT0000 complete [ 4331.830288] LustreError: 263225:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355512 with bad export cookie 8257597065794003441 [ 4331.833649] LustreError: 263225:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4331.833905] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4331.901461] LustreError: 267655:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4331.903562] LustreError: 267655:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4331.953630] Lustre: server umount lustre-MDT0001 complete [ 4335.861362] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 4336.877658] Key type lgssc unregistered [ 4337.009395] LNet: 268186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4337.011796] LNetError: 268186:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4337.017319] LNet: Removed LNI 192.168.202.129@tcp [ 4337.309356] Key type .llcrypt unregistered [ 4337.310392] Key type ._llcrypt unregistered [ 4346.680637] Key type ._llcrypt registered [ 4346.682320] Key type .llcrypt registered [ 4346.735513] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 4347.074827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4347.112939] alg: No test for adler32 (adler32-zlib) [ 4347.963716] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 4348.050237] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 4349.632154] Key type lgssc registered [ 4350.009766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4352.634364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4352.639554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4353.795977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4355.026096] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4357.693564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4357.811879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4359.087265] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4360.761328] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4361.979567] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4364.193658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4364.259358] LustreError: 271158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4364.266768] LustreError: 271158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4364.306402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4366.118628] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4367.898060] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4368.423530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4383.713912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4383.714906] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.716428] LustreError: Skipped 1 previous similar message [ 4383.720672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4383.723233] Lustre: Skipped 1 previous similar message [ 4387.438357] LustreError: 272579:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4387.473982] Lustre: server umount lustre-OST0000 complete [ 4388.834536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4388.834701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4388.835369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4388.842110] Lustre: Skipped 2 previous similar messages [ 4393.953229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4393.954927] Lustre: Skipped 2 previous similar messages [ 4394.814853] LustreError: 272781:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4394.818240] LustreError: 272781:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4394.848872] Lustre: server umount lustre-MDT0000 complete [ 4396.108161] LustreError: 271772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355576 with bad export cookie 17137138467770387595 [ 4396.111053] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4396.113744] LustreError: 271772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4396.184398] LustreError: 272983:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4396.187233] LustreError: 272983:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4396.235512] Lustre: server umount lustre-MDT0001 complete [ 4399.967736] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 4401.050887] Key type lgssc unregistered [ 4401.184425] LNet: 273514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4401.188613] LNetError: 273514:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4401.202395] LNet: Removed LNI 192.168.202.129@tcp [ 4401.531230] Key type .llcrypt unregistered [ 4401.532929] Key type ._llcrypt unregistered [ 4408.443589] Key type ._llcrypt registered [ 4408.445135] Key type .llcrypt registered [ 4408.481210] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 4408.980950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4409.028415] alg: No test for adler32 (adler32-zlib) [ 4409.907388] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 4410.005343] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 4411.592169] Key type lgssc registered [ 4412.040146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4414.681148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4414.686621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4415.843039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4417.142290] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4419.985169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4420.106836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4421.392771] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4423.162864] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4424.427474] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4426.725018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4426.851960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.794379] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4430.803724] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4441.192067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4445.665609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4445.667857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4445.672353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4446.689693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4446.690661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4446.691892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.698027] Lustre: Skipped 1 previous similar message [ 4450.799096] LustreError: 277107:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4450.834721] Lustre: server umount lustre-OST0000 complete [ 4455.904818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4455.906671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4455.909915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4458.172896] LustreError: 277308:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4458.174780] LustreError: 277308:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4458.201673] Lustre: server umount lustre-MDT0000 complete [ 4459.423110] LustreError: 274673:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355639 with bad export cookie 2287468498053334205 [ 4459.426593] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4459.428528] LustreError: 274673:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4459.506033] LustreError: 277509:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4459.508743] LustreError: 277509:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4459.560428] Lustre: server umount lustre-MDT0001 complete [ 4462.991326] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 4463.979883] Key type lgssc unregistered [ 4464.117432] LNet: 278039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4464.119716] LNetError: 278039:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4464.128430] LNet: Removed LNI 192.168.202.129@tcp [ 4464.423603] Key type .llcrypt unregistered [ 4464.424942] Key type ._llcrypt unregistered [ 4473.397981] Key type ._llcrypt registered [ 4473.399781] Key type .llcrypt registered [ 4473.437386] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 4473.821558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4473.850704] alg: No test for adler32 (adler32-zlib) [ 4474.704214] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 4474.781082] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 4476.360115] Key type lgssc registered [ 4476.721858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4479.163920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4479.169693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4480.317226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4481.550189] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4484.232188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4484.348239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4485.638071] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4487.326549] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4488.524224] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4490.723348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4490.828473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4492.843332] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4494.488435] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4506.215665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4519.904821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4519.907182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4519.912621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4521.956249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4521.956529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4521.960054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4523.375818] LustreError: 282469:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4523.418699] Lustre: server umount lustre-OST0000 complete [ 4525.537039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4525.539595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4525.543548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4525.545335] Lustre: Skipped 1 previous similar message [ 4530.812995] LustreError: 282671:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4530.814812] LustreError: 282671:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4530.839903] Lustre: server umount lustre-MDT0000 complete [ 4532.069261] LustreError: 279395:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355712 with bad export cookie 955424665785694274 [ 4532.072206] LustreError: 279395:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4532.072750] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4532.135625] LustreError: 282875:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4532.137352] LustreError: 282875:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4532.185600] Lustre: server umount lustre-MDT0001 complete [ 4535.499404] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 4536.493636] Key type lgssc unregistered [ 4536.619377] LNet: 283405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4536.622488] LNetError: 283405:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4536.631291] LNet: Removed LNI 192.168.202.129@tcp [ 4536.939479] Key type .llcrypt unregistered [ 4536.940629] Key type ._llcrypt unregistered [ 4543.721175] Key type ._llcrypt registered [ 4543.722224] Key type .llcrypt registered [ 4543.762277] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 4544.114832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4544.147859] alg: No test for adler32 (adler32-zlib) [ 4545.019588] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 4545.101793] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 4546.688127] Key type lgssc registered [ 4547.108074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4549.837243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4549.847436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4551.045604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4552.488992] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4555.630282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4555.765938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4557.107504] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4558.958757] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4560.120784] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4562.343576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4562.459256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4564.295760] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4566.100499] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4575.784318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4580.834744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4580.835776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4580.835854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4580.840846] Lustre: Skipped 1 previous similar message [ 4582.061160] LustreError: 287033:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4582.096358] Lustre: server umount lustre-OST0000 complete [ 4585.405648] LustreError: 287235:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4585.408497] LustreError: 287235:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4585.439813] Lustre: server umount lustre-MDT0000 complete [ 4585.953676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4585.954340] LustreError: 284592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4585.956818] Lustre: Skipped 1 previous similar message [ 4585.960473] LustreError: 284592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4586.638147] LustreError: 286241:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355767 with bad export cookie 26470963752540783 [ 4586.640587] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4586.642093] LustreError: 286241:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4586.704761] LustreError: 287435:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4586.706879] LustreError: 287435:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4586.760201] Lustre: server umount lustre-MDT0001 complete [ 4590.116132] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 4591.133567] Key type lgssc unregistered [ 4591.255465] LNet: 287966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4591.257893] LNetError: 287966:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4591.266382] LNet: Removed LNI 192.168.202.129@tcp [ 4591.592894] Key type .llcrypt unregistered [ 4591.593923] Key type ._llcrypt unregistered [ 4595.172930] Key type ._llcrypt registered [ 4595.173822] Key type .llcrypt registered [ 4595.212636] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4600.709753] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 4606.634864] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 4606.988872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4607.038033] alg: No test for adler32 (adler32-zlib) [ 4607.881140] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 4607.955038] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 4609.536140] Key type lgssc registered [ 4609.875645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4612.807852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4614.930958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4616.938377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4619.107121] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4621.321199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4621.342486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4621.346205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4622.426298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4622.436742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4622.475822] Lustre: lustre-MDT0000: new disk, initializing [ 4622.499774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4622.505899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4623.767140] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4627.771593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4627.793690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4627.828298] Lustre: 291450:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4627.851581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4627.853026] Lustre: Skipped 1 previous similar message [ 4627.887955] Lustre: lustre-MDT0001: new disk, initializing [ 4627.921371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4627.936030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4627.939700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4629.208919] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4632.219393] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4633.377040] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4635.645718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4635.669785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4635.768351] Lustre: lustre-OST0000: new disk, initializing [ 4635.770255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4635.803126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4636.988786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4636.993770] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4637.038041] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4637.813870] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4640.859802] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4642.144445] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4642.220148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4643.519339] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4643.591683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4647.395257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4647.397447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4647.398939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4647.401029] Lustre: Skipped 1 previous similar message [ 4650.349078] LustreError: 294389:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4650.382824] Lustre: server umount lustre-OST0000 complete [ 4652.515040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4652.515978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4652.518266] Lustre: Skipped 1 previous similar message [ 4652.520116] Lustre: Skipped 1 previous similar message [ 4657.633660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4657.636040] Lustre: Skipped 1 previous similar message [ 4657.733326] LustreError: 294590:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4657.735772] LustreError: 294590:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4657.762499] Lustre: server umount lustre-MDT0000 complete [ 4659.040341] LustreError: 292231:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355839 with bad export cookie 468551016350251028 [ 4659.043173] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4659.045815] LustreError: 292231:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4659.108941] LustreError: 294793:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4659.112646] LustreError: 294793:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4659.166549] Lustre: server umount lustre-MDT0001 complete [ 4661.896322] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 00:04:00 (1763355840) [ 4663.944723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4664.114852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4665.368583] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4668.214256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4668.322390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4669.652372] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4671.304765] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4672.442917] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4674.649634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4676.673589] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4678.450946] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4698.593022] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4698.596376] LustreError: Skipped 1 previous similar message [ 4698.598830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4698.606217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4699.618549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4703.022842] LustreError: 297735:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4703.024826] LustreError: 297735:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4703.064718] Lustre: server umount lustre-OST0000 complete [ 4704.737121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4704.743914] Lustre: Skipped 1 previous similar message [ 4704.746768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4704.748643] Lustre: Skipped 2 previous similar messages [ 4707.810048] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4710.591085] LustreError: 297936:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4710.593387] LustreError: 297936:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4710.618848] Lustre: server umount lustre-MDT0000 complete [ 4711.849105] LustreError: 295442:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355892 with bad export cookie 468551016350251896 [ 4711.851632] LustreError: 296183:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4711.853337] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4711.853419] LustreError: 295442:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4711.967612] Lustre: server umount lustre-MDT0001 complete [ 4715.305209] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 4716.314991] Key type lgssc unregistered [ 4716.433374] LNet: 298668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4716.435465] LNetError: 298668:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4716.442498] LNet: Removed LNI 192.168.202.129@tcp [ 4716.716886] Key type .llcrypt unregistered [ 4716.717933] Key type ._llcrypt unregistered [ 4724.910810] Key type ._llcrypt registered [ 4724.912754] Key type .llcrypt registered [ 4724.956411] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 4731.188952] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 4731.549558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4731.587696] alg: No test for adler32 (adler32-zlib) [ 4732.440110] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 4732.519966] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 4734.096184] Key type lgssc registered [ 4734.433863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4737.146247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4739.130174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4741.012289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4742.999354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4744.553027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4746.160059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4748.300199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4748.317662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4748.322211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4749.402337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4749.412292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4749.447663] Lustre: lustre-MDT0000: new disk, initializing [ 4749.471298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4749.477880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4750.703740] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4754.745712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4754.774410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4754.808977] Lustre: 302444:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4754.832493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4754.834135] Lustre: Skipped 1 previous similar message [ 4754.868474] Lustre: lustre-MDT0001: new disk, initializing [ 4754.899035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4754.910243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4754.913188] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4756.098205] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4759.067935] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4760.206674] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4762.554805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4762.581658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4762.675685] Lustre: lustre-OST2710: new disk, initializing [ 4762.677809] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4762.706816] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4764.501164] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4765.049319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4765.052514] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4765.087282] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4767.490021] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4769.688184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4769.708242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4769.758974] Lustre: lustre-OST03e8: new disk, initializing [ 4769.761728] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4769.795167] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4771.566519] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4773.187857] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4774.403354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4774.406045] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4774.439394] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4775.718832] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4775.793607] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4777.016345] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4777.092526] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4778.305920] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4778.370573] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4779.586511] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4779.648082] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4784.611919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4784.612883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4784.615019] Lustre: Skipped 1 previous similar message [ 4786.687493] LustreError: 306787:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4786.717202] Lustre: server umount lustre-MDT0000 complete [ 4787.926224] LustreError: 303226:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355968 with bad export cookie 15628467154649897358 [ 4787.929412] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4787.930726] LustreError: 303226:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4787.997558] LustreError: 306990:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4787.999175] LustreError: 306990:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4788.046700] Lustre: server umount lustre-MDT0001 complete [ 4798.899094] LustreError: 307191:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4798.900684] LustreError: 307191:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4798.915919] Lustre: server umount lustre-OST2710 complete [ 4810.292158] LustreError: 307392:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4810.294041] LustreError: 307392:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4810.335324] Lustre: server umount lustre-OST03e8 complete [ 4815.869388] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 4818.706269] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 4821.687575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4823.620185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4825.494293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4827.431602] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4829.572070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4829.592440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4829.669712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4829.679236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4829.714634] Lustre: lustre-MDT0000: new disk, initializing [ 4829.736422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4829.742905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4830.883505] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4835.202174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4835.235677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4835.282929] Lustre: 310183:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4835.409922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4836.788055] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4840.069663] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4841.769104] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4844.214713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4844.238209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4844.336855] Lustre: lustre-OST0000: new disk, initializing [ 4844.338118] Lustre: Skipped 1 previous similar message [ 4844.339620] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4844.341251] Lustre: Skipped 2 previous similar messages [ 4844.363662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4844.366380] Lustre: Skipped 1 previous similar message [ 4846.137832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4846.140061] Lustre: Skipped 1 previous similar message [ 4846.141715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4846.175419] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4846.199621] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4849.118789] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4850.355567] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4850.417517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4851.605496] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4851.673764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4856.290073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4856.290497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4856.292285] LustreError: Skipped 1 previous similar message [ 4856.293140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4856.295248] Lustre: Skipped 3 previous similar messages [ 4856.299329] Lustre: Skipped 4 previous similar messages [ 4858.348087] LustreError: 313118:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4858.349954] LustreError: 313118:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4858.386090] Lustre: server umount lustre-OST0000 complete [ 4860.896811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4860.898733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4860.901961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4865.686221] Lustre: server umount lustre-MDT0000 complete [ 4866.857124] LustreError: 310963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356047 with bad export cookie 15628467154649899024 [ 4866.860446] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4866.861823] LustreError: 310963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4866.929596] LustreError: 313521:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4866.931977] LustreError: 313521:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4866.980653] Lustre: server umount lustre-MDT0001 complete [ 4869.603456] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 00:07:28 (1763356048) [ 4870.056331] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4870.583679] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 00:07:29 (1763356049) [ 4871.431683] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 4873.911762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4875.168651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4876.519387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4877.920566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4878.341655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4880.443344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4880.530922] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4880.539936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4880.596662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4881.758722] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4884.386523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4884.417263] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4884.426141] Lustre: 315646:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4885.675757] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4887.234639] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4888.371879] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4890.558731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4890.583267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4890.647884] Lustre: Denying initial registration attempt from nid 192.168.202.129@tcp, specified as failover [ 4890.650509] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4890.653448] LustreError: 317263:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 4890.655254] LustreError: 317263:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4890.658294] LustreError: 317263:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4890.678737] Lustre: server umount lustre-OST0000 complete [ 4890.679815] LustreError: 317263:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4894.689023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4894.692340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4894.697787] Lustre: Skipped 1 previous similar message [ 4894.700280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4894.703111] Lustre: Skipped 2 previous similar messages [ 4898.881079] LustreError: 317630:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4898.883632] LustreError: 317630:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4898.909289] Lustre: server umount lustre-MDT0000 complete [ 4900.068073] LustreError: 315637:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356080 with bad export cookie 15628467154649899892 [ 4900.070487] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4900.070925] LustreError: 315637:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4900.168244] Lustre: server umount lustre-MDT0001 complete [ 4903.413160] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 4904.343768] Key type lgssc unregistered [ 4904.470338] LNet: 318363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4904.472770] LNetError: 318363:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4904.480323] LNet: Removed LNI 192.168.202.129@tcp [ 4904.739832] Key type .llcrypt unregistered [ 4904.740792] Key type ._llcrypt unregistered [ 4907.864970] Key type ._llcrypt registered [ 4907.865940] Key type .llcrypt registered [ 4907.896897] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 4908.182021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4908.218041] alg: No test for adler32 (adler32-zlib) [ 4909.079332] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 4909.160700] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 4910.744104] Key type lgssc registered [ 4911.120691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4913.458327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4914.799356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4916.041811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4917.340566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4917.672685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4923.647987] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 4926.326556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4926.334066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4927.423475] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4927.432679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4927.502466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4929.109471] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4931.777454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4931.805922] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4931.819976] Lustre: 321093:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4931.911291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4933.067545] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4934.853717] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4935.934778] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4938.834648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4938.875702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4938.972790] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4939.036844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4940.963275] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4942.736294] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4945.891607] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4945.891948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4945.892732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4945.899144] Lustre: Skipped 1 previous similar message [ 4949.548975] LustreError: 323376:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4949.580314] Lustre: server umount lustre-OST0000 complete [ 4951.010284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4951.011479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4951.013206] Lustre: Skipped 1 previous similar message [ 4951.015932] Lustre: Skipped 2 previous similar messages [ 4956.129231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4956.131450] Lustre: Skipped 1 previous similar message [ 4961.249175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4961.251847] Lustre: Skipped 1 previous similar message [ 4965.344087] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4965.379875] LustreError: 323577:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4965.382381] LustreError: 323577:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4965.413256] Lustre: server umount lustre-MDT0000 complete [ 4966.571298] LustreError: 321085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356146 with bad export cookie 15461582071604808834 [ 4966.574720] LustreError: 321085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4966.575039] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4966.643345] LustreError: 323781:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4966.646584] LustreError: 323781:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4966.696279] Lustre: server umount lustre-MDT0001 complete [ 4969.890918] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 4970.842561] Key type lgssc unregistered [ 4970.966336] LNet: 324312:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4970.968349] LNetError: 324312:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4970.977303] LNet: Removed LNI 192.168.202.129@tcp [ 4971.271964] Key type .llcrypt unregistered [ 4971.273541] Key type ._llcrypt unregistered [ 4979.536132] Key type ._llcrypt registered [ 4979.537062] Key type .llcrypt registered [ 4979.569255] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 4979.910930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4979.940202] alg: No test for adler32 (adler32-zlib) [ 4980.795452] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 4980.883052] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 4982.464116] Key type lgssc registered [ 4982.837341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4985.330788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4985.337048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4986.479274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4987.628173] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4990.262403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4990.376869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4991.566511] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 4993.325528] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4994.507385] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4996.628731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4996.758286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4998.628013] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5000.485236] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5016.032911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5016.034659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5016.037762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5016.546661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5016.547297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5016.549465] Lustre: Skipped 1 previous similar message [ 5016.551779] Lustre: Skipped 2 previous similar messages [ 5019.271905] LustreError: 327952:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5019.300341] Lustre: server umount lustre-MDT0000 complete [ 5020.557392] LustreError: 325665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356200 with bad export cookie 4786054474815458302 [ 5020.560801] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5020.562505] LustreError: 325665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5021.401987] LustreError: 325679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5021.410401] LustreError: 325679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5021.410909] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.29@tcp (stopping) [ 5021.665601] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5025.696861] LustreError: 327089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5025.697088] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5025.701445] LustreError: 327089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5025.702794] Lustre: Skipped 1 previous similar message [ 5026.684269] LustreError: 328157:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5026.685872] LustreError: 328157:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5026.735584] Lustre: server umount lustre-MDT0001 complete [ 5028.888491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5031.628345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5045.216956] LustreError: 324995:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002ba8aaa0 x1849012949900928/t0(0) o250->MGC192.168.202.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5045.339855] LustreError: 327292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5045.346209] LustreError: 327292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5045.369325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5045.394372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 5046.651376] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5049.393984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5049.516473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5050.533269] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5050.688465] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5052.424116] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5054.628574] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5057.067726] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5058.239250] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5058.776599] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5060.069967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5060.070875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5060.071784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5060.071789] Lustre: Skipped 1 previous similar message [ 5060.075681] Lustre: Skipped 1 previous similar message [ 5066.097461] LustreError: 330458:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5066.099229] LustreError: 330458:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5066.139965] Lustre: server umount lustre-OST0000 complete [ 5070.306990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5070.307042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5070.315441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5070.317110] Lustre: Skipped 4 previous similar messages [ 5073.470462] LustreError: 330660:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5073.472301] LustreError: 330660:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5073.500600] Lustre: server umount lustre-MDT0000 complete [ 5074.726507] LustreError: 327303:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356255 with bad export cookie 4786054474815465456 [ 5074.731990] LustreError: 327303:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5074.732451] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5074.853957] Lustre: server umount lustre-MDT0001 complete [ 5078.195565] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 5079.156381] Key type lgssc unregistered [ 5079.273256] LNet: 331393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5079.275362] LNetError: 331393:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5079.283304] LNet: Removed LNI 192.168.202.129@tcp [ 5079.602426] Key type .llcrypt unregistered [ 5079.604132] Key type ._llcrypt unregistered [ 5088.266103] Key type ._llcrypt registered [ 5088.266981] Key type .llcrypt registered [ 5088.297487] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 5088.576282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5088.581846] alg: No test for adler32 (adler32-zlib) [ 5089.432594] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 5089.514802] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 5091.088100] Key type lgssc registered [ 5091.410203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5094.079144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5094.089530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5095.266230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5096.393802] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5099.263971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5099.410246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5100.634715] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5102.290188] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5103.359619] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5104.427796] LustreError: 332794:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5104.430669] LustreError: 332794:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5104.432965] LustreError: 332794:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 5104.609474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5104.609480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5104.610378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5104.617877] Lustre: Skipped 1 previous similar message [ 5108.706429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5108.710259] Lustre: Skipped 1 previous similar message [ 5110.549127] LustreError: 334156:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5110.584570] Lustre: server umount lustre-MDT0000 complete [ 5111.818241] LustreError: 333484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356292 with bad export cookie 6930962187382632776 [ 5111.821115] LustreError: 333484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5111.821502] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5111.881048] LustreError: 334360:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5111.882669] LustreError: 334360:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5111.930575] Lustre: server umount lustre-MDT0001 complete [ 5117.438797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5117.529560] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5117.539107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5117.608731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5118.876727] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5121.989199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5122.023903] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5122.033339] Lustre: 335129:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5122.125145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5123.299969] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5125.062565] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5126.242186] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5129.210511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5129.271960] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5129.274636] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5129.276884] LustreError: 336789:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 5129.278725] LustreError: 336789:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 5129.280274] LustreError: 336789:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 5129.298342] Lustre: server umount lustre-OST0000 complete [ 5129.299359] LustreError: 336789:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5132.524822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5132.587570] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5132.651332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5133.582721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 5134.385084] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5136.094424] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5138.646189] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5138.669569] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5138.697317] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5138.739574] Lustre: lustre-OST0001: new disk, initializing [ 5138.742936] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5140.564903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5140.569381] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5140.604816] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5141.175103] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5142.856813] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5146.090313] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5146.090573] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5146.091989] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5146.095100] LustreError: Skipped 1 previous similar message [ 5146.107902] Lustre: Skipped 2 previous similar messages [ 5149.876174] LustreError: 338905:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5149.880114] LustreError: 338905:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5149.909445] Lustre: server umount lustre-OST0001 complete [ 5151.204963] LustreError: 337257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5151.206131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5151.207332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5151.207339] Lustre: Skipped 1 previous similar message [ 5151.213255] LustreError: 337257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5151.232252] Lustre: Skipped 1 previous similar message [ 5156.323340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5156.324284] LustreError: 339109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5156.324862] Lustre: Skipped 2 previous similar messages [ 5156.331979] LustreError: 339109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5157.169186] LustreError: 339108:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5157.174450] LustreError: 339108:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5157.218461] Lustre: server umount lustre-OST0000 complete [ 5161.443594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5161.446613] Lustre: Skipped 1 previous similar message [ 5164.554058] LustreError: 339310:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5164.558117] LustreError: 339310:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5164.598621] Lustre: server umount lustre-MDT0000 complete [ 5165.932477] LustreError: 335854:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356346 with bad export cookie 6930962187382633308 [ 5165.935858] LustreError: 335854:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5165.936043] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5166.064074] Lustre: server umount lustre-MDT0001 complete [ 5169.502596] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 5170.530260] Key type lgssc unregistered [ 5170.672557] LNet: 340042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5170.676650] LNetError: 340042:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5170.684573] LNet: Removed LNI 192.168.202.129@tcp [ 5170.986308] Key type .llcrypt unregistered [ 5170.987290] Key type ._llcrypt unregistered [ 5182.431222] Key type ._llcrypt registered [ 5182.433438] Key type .llcrypt registered [ 5182.468631] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 5188.506241] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 5188.906771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5188.920645] alg: No test for adler32 (adler32-zlib) [ 5189.769454] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 5189.850873] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 5191.432130] Key type lgssc registered [ 5191.768923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5194.607073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5196.673852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5198.592267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5200.651410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5202.976393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5202.998288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5203.002763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5204.091178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5204.106219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5204.150039] Lustre: lustre-MDT0000: new disk, initializing [ 5204.176113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5204.183235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5205.398265] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5209.222260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5209.246039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5209.277100] Lustre: 343964:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5209.299683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5209.302386] Lustre: Skipped 1 previous similar message [ 5209.345562] Lustre: lustre-MDT0001: new disk, initializing [ 5209.378174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5209.389154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5209.393458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5210.590957] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5213.508350] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5214.600308] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5216.714711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5216.734672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5216.820776] Lustre: lustre-OST0000: new disk, initializing [ 5216.822476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5216.847541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5218.628892] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5221.575637] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5222.806616] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5223.415094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5223.418794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5223.444818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5223.886663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5225.030783] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5225.092773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5228.516346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5228.516535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5228.518289] LustreError: Skipped 1 previous similar message [ 5228.518948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5228.521170] Lustre: Skipped 1 previous similar message [ 5231.727118] LustreError: 346908:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5231.763575] Lustre: server umount lustre-OST0000 complete [ 5233.633841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5233.634567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5233.637029] Lustre: Skipped 1 previous similar message [ 5233.639017] Lustre: Skipped 1 previous similar message [ 5238.753393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5238.755091] Lustre: Skipped 2 previous similar messages [ 5239.040193] LustreError: 347109:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5239.041746] LustreError: 347109:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5239.071441] Lustre: server umount lustre-MDT0000 complete [ 5240.274040] LustreError: 343955:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356420 with bad export cookie 13002191647050859811 [ 5240.277477] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5240.277712] LustreError: 343955:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5240.338382] LustreError: 347310:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5240.341197] LustreError: 347310:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5240.390465] Lustre: server umount lustre-MDT0001 complete [ 5242.922918] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 00:13:41 (1763356421) [ 5244.612508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5246.341165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5253.152249] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 5255.782703] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 5258.649424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5260.671690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5262.556206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5264.531323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5266.681383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5266.702200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5266.776517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5266.785936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5266.820942] Lustre: lustre-MDT0000: new disk, initializing [ 5266.843154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5266.848825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5268.021542] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5271.877649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5271.899620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5271.927892] Lustre: 351074:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5271.946874] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5271.948267] Lustre: Skipped 1 previous similar message [ 5271.981639] Lustre: lustre-MDT0001: new disk, initializing [ 5272.010932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5272.020669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5272.024408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5273.161158] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5276.036576] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5277.143976] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5279.162311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5279.184659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5280.994547] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5283.951198] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5285.218479] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5285.369937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5285.372771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5285.403882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5286.295106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5287.422402] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5287.487742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5290.466842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5290.467359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5290.471162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5290.473172] LustreError: Skipped 1 previous similar message [ 5294.127096] LustreError: 354019:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5294.130527] LustreError: 354019:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5294.168961] Lustre: server umount lustre-OST0000 complete [ 5295.585378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5295.586027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5295.593271] Lustre: Skipped 2 previous similar messages [ 5295.594701] Lustre: Skipped 1 previous similar message [ 5301.438321] LustreError: 354220:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5301.442794] LustreError: 354220:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5301.475371] Lustre: server umount lustre-MDT0000 complete [ 5302.603982] LustreError: 351067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356482 with bad export cookie 13002191647050860679 [ 5302.606795] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5302.606885] LustreError: 351067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5302.718192] Lustre: server umount lustre-MDT0001 complete [ 5305.293507] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:14:44 (1763356484) [ 5307.511217] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:14:46 (1763356486) [ 5310.164561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5310.340060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5310.341881] Lustre: Skipped 1 previous similar message [ 5311.561092] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5314.154086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5315.465464] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5317.074456] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5318.103987] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5320.174269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5321.977964] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5323.631858] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5327.866757] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5328.399986] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5328.893878] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5329.395086] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5330.008542] Lustre: Failing over lustre-MDT0000 [ 5330.089932] LustreError: 357876:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5330.091469] LustreError: 357876:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5330.113857] Lustre: server umount lustre-MDT0000 complete [ 5330.143920] LustreError: 356154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5330.147923] LustreError: 356154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5335.009613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5335.011309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5335.012192] LustreError: 355434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5335.016917] Lustre: Skipped 2 previous similar messages [ 5335.020708] LustreError: 355434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5340.129551] LustreError: 355434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5340.136614] LustreError: 355434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5342.835421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5342.877886] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5342.967809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5342.970289] Lustre: Skipped 2 previous similar messages [ 5342.995200] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5344.148948] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5345.497934] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5348.332832] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5348.334823] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5348.351736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5349.429278] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5349.941451] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5351.691297] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5352.288115] Lustre: Failing over lustre-MDT0000 [ 5352.445145] LustreError: 358955:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5352.446710] LustreError: 358955:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5352.482170] Lustre: server umount lustre-MDT0000 complete [ 5353.443231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5353.444207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5353.445580] LustreError: 355428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5353.450027] Lustre: Skipped 1 previous similar message [ 5353.453822] LustreError: 355428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5358.561764] LustreError: 356147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5358.567977] LustreError: 356147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5365.234319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5365.267600] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5365.383333] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5365.979770] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5366.588814] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5370.858994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5370.862197] Lustre: Skipped 2 previous similar messages [ 5370.868471] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5370.885289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5371.983521] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5372.450976] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5375.971134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5375.971583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5375.977089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5375.978082] Lustre: Skipped 3 previous similar messages [ 5375.979695] Lustre: Skipped 4 previous similar messages [ 5380.758441] Lustre: server umount lustre-MDT0000 complete [ 5381.089522] LustreError: 355433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5381.093068] LustreError: 355433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5381.897500] LustreError: 355459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356562 with bad export cookie 13002191647050863318 [ 5381.902772] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5382.020261] Lustre: server umount lustre-MDT0001 complete [ 5393.457411] LustreError: 360385:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5393.459010] LustreError: 360385:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5393.493219] Lustre: server umount lustre-OST0000 complete [ 5398.133831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5398.272228] LustreError: 361212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5398.293697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5398.295735] Lustre: Skipped 1 previous similar message [ 5399.437286] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5402.141751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5403.492781] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5405.147242] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5406.211700] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5408.353590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5409.511370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5410.215681] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5411.867593] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5420.000911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5420.003616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5420.006936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5420.008475] Lustre: Skipped 1 previous similar message [ 5424.220532] Lustre: server umount lustre-OST0000 complete [ 5431.514070] Lustre: server umount lustre-MDT0000 complete [ 5432.668082] LustreError: 361194:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356613 with bad export cookie 13002191647050864046 [ 5432.670899] LustreError: 361194:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5432.671189] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5432.783243] Lustre: server umount lustre-MDT0001 complete [ 5435.998212] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 5436.907654] Key type lgssc unregistered [ 5437.019287] LNet: 364390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5437.021303] LNetError: 364390:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5437.034270] LNet: Removed LNI 192.168.202.129@tcp [ 5437.327030] Key type .llcrypt unregistered [ 5437.328878] Key type ._llcrypt unregistered [ 5445.895413] Key type ._llcrypt registered [ 5445.896296] Key type .llcrypt registered [ 5445.929761] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 5446.274441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5446.333369] alg: No test for adler32 (adler32-zlib) [ 5447.191795] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 5447.273059] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 5448.856135] Key type lgssc registered [ 5449.250264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5451.856892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5451.863267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5453.026158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5454.232104] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5456.850702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5456.959780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5458.067716] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5459.715663] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5460.926156] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5463.065688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5463.164148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5464.823293] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5466.410385] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5478.568416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5487.010648] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5487.011741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5487.013507] Lustre: Skipped 2 previous similar messages [ 5487.015860] Lustre: Skipped 1 previous similar message [ 5492.092438] LustreError: 368131:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5492.117787] Lustre: server umount lustre-MDT0000 complete [ 5492.193668] LustreError: 366575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5492.199571] LustreError: 366575:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5493.237858] LustreError: 365844:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356673 with bad export cookie 15176126406841895219 [ 5493.240530] LustreError: 365844:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5493.240778] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5493.306435] LustreError: 368334:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5493.308237] LustreError: 368334:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5493.347398] Lustre: server umount lustre-MDT0001 complete [ 5503.729808] LustreError: 368535:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5503.731429] LustreError: 368535:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5503.765599] Lustre: server umount lustre-OST0000 complete [ 5507.723308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5507.857632] LustreError: 369264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5507.877796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5509.019807] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5511.631858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5512.935687] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5514.560406] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5515.625175] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5517.767345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5517.797333] LustreError: 370862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5517.878556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5517.881729] Lustre: Skipped 1 previous similar message [ 5519.682761] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5521.309464] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5521.963832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5537.251626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5537.251941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5537.253778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5537.253786] Lustre: Skipped 1 previous similar message [ 5537.254582] Lustre: Skipped 1 previous similar message [ 5539.390883] LustreError: 371612:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5539.393959] LustreError: 371612:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5539.426937] Lustre: server umount lustre-MDT0000 complete [ 5540.579418] LustreError: 369244:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356720 with bad export cookie 15176126406841896325 [ 5540.583531] LustreError: 369244:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5540.584794] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5540.700923] Lustre: server umount lustre-MDT0001 complete [ 5552.117776] LustreError: 372016:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5552.120643] LustreError: 372016:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5552.161563] Lustre: server umount lustre-OST0000 complete [ 5556.965660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5557.106850] LustreError: 372843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5557.134766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5558.344164] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5561.116748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5562.465777] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5564.185621] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5565.287900] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5567.421032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5567.460035] LustreError: 374440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5569.169412] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5570.784680] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5571.057599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5581.280920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5581.284883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5581.289752] Lustre: Skipped 1 previous similar message [ 5581.291861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5581.293638] Lustre: Skipped 2 previous similar messages [ 5582.307448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5583.469186] LustreError: 375088:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5583.472063] LustreError: 375088:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5583.507268] Lustre: server umount lustre-OST0000 complete [ 5586.912802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5586.914919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5586.917979] Lustre: Skipped 1 previous similar message [ 5586.919488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5586.922211] Lustre: Skipped 2 previous similar messages [ 5590.800605] Lustre: server umount lustre-MDT0000 complete [ 5591.892192] LustreError: 372823:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356772 with bad export cookie 15176126406841897333 [ 5591.894958] LustreError: 372823:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5591.895517] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5591.995185] Lustre: server umount lustre-MDT0001 complete [ 5595.301353] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 5596.150737] Key type lgssc unregistered [ 5596.271339] LNet: 376022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5596.273215] LNetError: 376022:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5596.284276] LNet: Removed LNI 192.168.202.129@tcp [ 5596.559446] Key type .llcrypt unregistered [ 5596.560394] Key type ._llcrypt unregistered [ 5605.645602] Key type ._llcrypt registered [ 5605.646613] Key type .llcrypt registered [ 5605.686531] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 5605.987166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5606.022649] alg: No test for adler32 (adler32-zlib) [ 5606.871799] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 5606.945912] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 5608.520097] Key type lgssc registered [ 5608.836336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5611.110591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5611.115327] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5611.118680] LustreError: 377420:0:(osd_handler.c:8577:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5611.128877] LustreError: 377420:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5611.130613] LustreError: 377420:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5611.132368] LustreError: 377420:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5611.134331] LustreError: 377420:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5613.385896] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5613.389163] LustreError: 377818:0:(osd_handler.c:8577:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5613.398765] LustreError: 377818:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5613.401767] LustreError: 377818:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5613.404797] LustreError: 377818:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5613.408255] LustreError: 377818:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5618.888660] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 5620.189615] Key type lgssc unregistered [ 5620.300364] LNet: 378801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5620.304600] LNetError: 378801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5620.319290] LNet: Removed LNI 192.168.202.129@tcp [ 5620.565578] Key type .llcrypt unregistered [ 5620.566884] Key type ._llcrypt unregistered [ 5625.949153] Key type ._llcrypt registered [ 5625.949948] Key type .llcrypt registered [ 5625.987484] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 5631.899787] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 5632.221175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5632.268096] alg: No test for adler32 (adler32-zlib) [ 5633.108718] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 5633.182094] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 5634.752113] Key type lgssc registered [ 5635.043558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5637.663851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5639.655712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5641.540833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5643.491563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5645.589047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5645.616939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5645.620650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5646.699725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5646.709927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5646.745964] Lustre: lustre-MDT0000: new disk, initializing [ 5646.768188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5646.774816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5647.914798] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5651.819862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5651.841321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5651.873380] Lustre: 381905:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5651.899486] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5651.902069] Lustre: Skipped 1 previous similar message [ 5651.942995] Lustre: lustre-MDT0001: new disk, initializing [ 5651.972323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5651.983562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5651.986814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5653.117319] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5655.973970] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5656.962246] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5658.977370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5659.000855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5659.084765] Lustre: lustre-OST0000: new disk, initializing [ 5659.086403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5659.113204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5659.892082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5659.896131] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5659.927534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5660.778572] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5663.665762] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5664.843545] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5664.907301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5666.125663] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5666.189059] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5670.370047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5670.371317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5670.372952] LustreError: Skipped 1 previous similar message [ 5670.378832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5670.382721] Lustre: Skipped 1 previous similar message [ 5672.878027] LustreError: 384840:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5672.917131] Lustre: server umount lustre-OST0000 complete [ 5675.489940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5675.490728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5675.493402] Lustre: Skipped 2 previous similar messages [ 5680.191327] LustreError: 385042:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5680.194022] LustreError: 385042:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5680.221369] Lustre: server umount lustre-MDT0000 complete [ 5681.381921] LustreError: 381895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356861 with bad export cookie 6426333985882574724 [ 5681.385174] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5681.386716] LustreError: 381895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5681.449502] LustreError: 385244:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5681.451652] LustreError: 385244:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5681.495646] Lustre: server umount lustre-MDT0001 complete [ 5684.011991] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5684.520699] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:21:03 (1763356863) [ 5686.505034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5686.645850] LustreError: 385962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5686.651836] LustreError: 385962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5686.667465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5687.823127] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5690.364328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5691.630558] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5693.191139] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5694.250409] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5696.267159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5696.365728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5696.367261] Lustre: Skipped 1 previous similar message [ 5698.019684] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5699.619275] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5701.704676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5701.730225] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5701.782603] Lustre: lustre-OST0001: new disk, initializing [ 5701.784126] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5703.360941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5703.363620] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5703.396522] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5703.488231] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5706.360067] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5708.771382] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5708.772865] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5708.774427] LustreError: Skipped 1 previous similar message [ 5708.779934] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5708.782063] Lustre: Skipped 2 previous similar messages [ 5712.608111] Lustre: lustre-OST0001: Not available for connect from 192.168.202.29@tcp (stopping) [ 5714.156832] LustreError: 389159:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5714.158464] LustreError: 389159:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5714.173809] Lustre: server umount lustre-OST0001 complete [ 5719.013608] LustreError: 387557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5719.017484] LustreError: 387557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5724.132716] LustreError: 387558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5724.137016] LustreError: 387558:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5726.177250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5726.179256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5726.182460] Lustre: Skipped 1 previous similar message [ 5726.183949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5726.185395] Lustre: Skipped 2 previous similar messages [ 5728.224765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5729.251967] LustreError: 387556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5729.255484] LustreError: 387556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5732.079976] LustreError: 389382:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5732.082407] LustreError: 389382:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5732.114260] Lustre: server umount lustre-OST0000 complete [ 5734.369842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5734.370620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5734.372589] Lustre: Skipped 2 previous similar messages [ 5734.373988] Lustre: Skipped 3 previous similar messages [ 5739.476691] Lustre: server umount lustre-MDT0000 complete [ 5739.488873] LustreError: 385961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5739.492932] LustreError: 385961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5740.591956] LustreError: 386678:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356920 with bad export cookie 6426333985882575592 [ 5740.595464] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5746.684104] LustreError: 389785:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5746.685785] LustreError: 389785:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5746.729113] Lustre: server umount lustre-MDT0001 complete [ 5749.682095] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 5750.564433] Key type lgssc unregistered [ 5750.681402] LNet: 390317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5750.684518] LNetError: 390317:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5750.695321] LNet: Removed LNI 192.168.202.129@tcp [ 5750.947893] Key type .llcrypt unregistered [ 5750.948786] Key type ._llcrypt unregistered [ 5760.681539] Key type ._llcrypt registered [ 5760.682446] Key type .llcrypt registered [ 5760.717306] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 5766.397877] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 5766.680391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5766.735491] alg: No test for adler32 (adler32-zlib) [ 5767.569390] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 5767.640239] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 5769.216088] Key type lgssc registered [ 5769.522918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5772.022352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5773.929466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5775.722421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5777.588334] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5779.588656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5779.602037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5779.605947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5780.681555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5780.691215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5780.727098] Lustre: lustre-MDT0000: new disk, initializing [ 5780.749940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5780.755667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5781.857460] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5785.575977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5785.594653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5785.623210] Lustre: 394237:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5785.643282] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5785.644716] Lustre: Skipped 1 previous similar message [ 5785.678452] Lustre: lustre-MDT0001: new disk, initializing [ 5785.707408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5785.718438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5785.722102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5786.832277] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5789.668624] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5790.744138] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5792.715032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5792.738591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5792.823936] Lustre: lustre-OST0000: new disk, initializing [ 5792.825622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5792.849559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5794.441290] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5797.267389] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5798.417114] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5798.901047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5798.903544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5798.932043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5799.490130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5800.595305] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5800.656919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5804.004053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5804.004553] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5804.004748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5804.006795] Lustre: Skipped 1 previous similar message [ 5804.008500] LustreError: Skipped 1 previous similar message [ 5807.278189] LustreError: 397180:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5807.310200] Lustre: server umount lustre-OST0000 complete [ 5809.121830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5809.122288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5809.124634] Lustre: Skipped 1 previous similar message [ 5809.128516] Lustre: Skipped 2 previous similar messages [ 5814.241044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5814.242498] Lustre: Skipped 1 previous similar message [ 5814.524098] LustreError: 397382:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5814.525633] LustreError: 397382:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5814.549528] Lustre: server umount lustre-MDT0000 complete [ 5815.612175] LustreError: 395017:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356996 with bad export cookie 13355137794176399808 [ 5815.615053] LustreError: 395017:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5815.615499] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5815.684865] LustreError: 397583:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5815.686476] LustreError: 397583:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5815.731087] Lustre: server umount lustre-MDT0001 complete [ 5818.250812] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 00:23:17 (1763356997) [ 5820.899478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5823.306716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5823.434119] Lustre: lustre-MDT0000: new disk, initializing [ 5823.462996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5824.534976] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5827.004274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5828.190669] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5829.745771] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5830.809416] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5831.405586] LustreError: 398875:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5831.410049] LustreError: 398875:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5831.413775] LustreError: 398875:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 5832.162412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5832.163231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5832.167271] Lustre: Skipped 1 previous similar message [ 5832.167552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5832.168807] Lustre: Skipped 1 previous similar message [ 5837.281055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5837.282680] Lustre: Skipped 1 previous similar message [ 5837.563959] LustreError: 400190:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5837.566603] LustreError: 400190:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5837.588322] Lustre: server umount lustre-MDT0000 complete [ 5838.703078] LustreError: 398828:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357019 with bad export cookie 13355137794176400676 [ 5838.706101] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5838.708423] LustreError: 398828:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5838.814716] Lustre: server umount lustre-MDT0001 complete [ 5841.633354] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:23:40 (1763357020) [ 5843.656565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5843.806152] LustreError: 401111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.810743] LustreError: 401111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5843.828392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.829931] Lustre: Skipped 1 previous similar message [ 5844.950528] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5847.494192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5848.677652] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5850.220269] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5851.286193] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5853.235426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5853.332281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5853.335469] Lustre: Skipped 1 previous similar message [ 5854.913036] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5856.446737] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5859.966455] Lustre: Permanently deactivating lustre-OST0000 [ 5859.971703] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5866.989927] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5868.103547] LustreError: 403847:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5868.106149] LustreError: 403847:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 5868.108082] LustreError: 403847:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5868.655858] LustreError: 403895:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 5868.657888] LustreError: 403895:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5875.310266] LustreError: 403992:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5875.311659] LustreError: 403992:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5875.338414] Lustre: server umount lustre-OST0000 complete [ 5876.043251] LustreError: 404146:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 5876.044970] LustreError: 404146:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5878.240673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5878.242451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5878.245161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5886.434757] LustreError: 404243:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5886.436082] LustreError: 404243:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5886.454683] Lustre: server umount lustre-MDT0000 complete [ 5887.436188] LustreError: 401090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357067 with bad export cookie 13355137794176401194 [ 5887.439109] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5887.440897] LustreError: 401090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5896.712358] Lustre: server umount lustre-MDT0001 complete [ 5898.975618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5900.030642] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5900.779843] LustreError: 405289:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5902.510757] Lustre: 405627:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.129@tcp [ 5903.065816] LustreError: 405724:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5903.067242] LustreError: 405724:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5903.088351] Lustre: server umount MGS complete [ 5906.215771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5906.362787] LustreError: 406351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5906.367312] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5906.368892] Lustre: Skipped 1 previous similar message [ 5906.385966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5907.403397] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5909.714468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5909.805120] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5910.844211] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5912.283131] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5913.215185] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5913.725455] Lustre: Permanently reactivating lustre-OST0000 [ 5913.728926] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5913.730319] Lustre: Skipped 2 previous similar messages [ 5940.192143] LustreError: 407957:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5942.927462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5943.027138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5943.028823] Lustre: Skipped 1 previous similar message [ 5944.590163] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 5946.060806] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5960.674278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5960.675197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5960.675863] LustreError: Skipped 1 previous similar message [ 5960.678229] Lustre: Skipped 1 previous similar message [ 5960.678624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5960.681301] Lustre: Skipped 5 previous similar messages [ 5963.373375] LustreError: 409832:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5963.374755] LustreError: 409832:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5963.404315] Lustre: server umount lustre-OST0000 complete [ 5970.506169] Lustre: server umount lustre-MDT0000 complete [ 5970.912880] LustreError: 406369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5970.916530] LustreError: 406369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5971.528431] LustreError: 406332:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357151 with bad export cookie 13355137794176402139 [ 5971.530632] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5971.531534] LustreError: 406332:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5971.632167] Lustre: server umount lustre-MDT0001 complete [ 5974.418212] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 5975.185297] Key type lgssc unregistered [ 5975.301334] LNet: 410768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5975.303728] LNetError: 410768:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5975.315360] LNet: Removed LNI 192.168.202.129@tcp [ 5975.563555] Key type .llcrypt unregistered [ 5975.564372] Key type ._llcrypt unregistered [ 5980.140832] Key type ._llcrypt registered [ 5980.141562] Key type .llcrypt registered [ 5980.168661] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 5985.444383] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 5985.745471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5985.774891] alg: No test for adler32 (adler32-zlib) [ 5986.611118] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 5986.683242] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 5988.256092] Key type lgssc registered [ 5988.540696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5990.927055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5992.695832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5994.404905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5996.190067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5998.156908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5998.172672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5998.176545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5999.253236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5999.262678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5999.298975] Lustre: lustre-MDT0000: new disk, initializing [ 5999.321062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5999.326641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6000.431797] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6004.126812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6004.146684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6004.175911] Lustre: 413870:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6004.195121] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6004.196617] Lustre: Skipped 1 previous similar message [ 6004.229871] Lustre: lustre-MDT0001: new disk, initializing [ 6004.258643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6004.268730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6004.271356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6005.352579] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6008.098623] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6009.113768] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6011.038392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6011.060779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6011.148943] Lustre: lustre-OST0000: new disk, initializing [ 6011.150550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6011.174579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6011.587306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6011.589884] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6011.623354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6012.767765] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6015.555024] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6016.695412] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6016.753282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6017.828591] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6017.886290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6020.462579] LustreError: 416806:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6020.496458] Lustre: server umount lustre-OST0000 complete [ 6022.113898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6022.114469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6022.116854] Lustre: Skipped 1 previous similar message [ 6027.233032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6027.235587] Lustre: Skipped 2 previous similar messages [ 6031.331662] LustreError: 417007:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6031.334210] LustreError: 417007:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6031.357113] Lustre: server umount lustre-MDT0000 complete [ 6032.482159] LustreError: 413862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357212 with bad export cookie 17733356284679206631 [ 6032.485218] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6032.485219] LustreError: 413862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6042.083800] LustreError: 417208:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6042.085304] LustreError: 417208:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6042.124667] Lustre: server umount lustre-MDT0001 complete [ 6044.523711] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6044.993821] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:27:03 (1763357223) [ 6046.993654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6047.131551] LustreError: 417929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6047.137692] LustreError: 417929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6047.153091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6048.275876] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6050.804403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6052.455960] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6054.048854] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6055.048494] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6057.011701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6057.108563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6057.110384] Lustre: Skipped 1 previous similar message [ 6058.668105] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6060.143780] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6060.689116] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6062.074973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6063.584669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6063.586659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6063.590164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6066.659633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6066.659904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6066.663823] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6069.228094] LustreError: 420364:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6069.230610] LustreError: 420364:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6069.260726] Lustre: server umount lustre-OST0000 complete [ 6071.777767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6071.778141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6071.778852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6071.778856] Lustre: Skipped 1 previous similar message [ 6071.784816] Lustre: Skipped 1 previous similar message [ 6076.412810] LustreError: 420569:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6076.414523] LustreError: 420569:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6076.439057] Lustre: server umount lustre-MDT0000 complete [ 6076.897027] LustreError: 417947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6076.901458] LustreError: 417947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6077.509038] LustreError: 417910:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357257 with bad export cookie 17733356284679207499 [ 6077.511688] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6077.513578] LustreError: 417910:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6077.618345] Lustre: server umount lustre-MDT0001 complete [ 6080.555190] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 6081.384366] Key type lgssc unregistered [ 6081.496295] LNet: 421302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6081.498104] LNetError: 421302:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6081.506293] LNet: Removed LNI 192.168.202.129@tcp [ 6081.747268] Key type .llcrypt unregistered [ 6081.748151] Key type ._llcrypt unregistered [ 6088.236169] Key type ._llcrypt registered [ 6088.237346] Key type .llcrypt registered [ 6088.267505] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 6088.347272] Key type .llcrypt unregistered [ 6088.348138] Key type ._llcrypt unregistered [ 6094.436816] Key type ._llcrypt registered [ 6094.437728] Key type .llcrypt registered [ 6094.472269] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 6094.853216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6094.905215] alg: No test for adler32 (adler32-zlib) [ 6095.764410] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 6095.845181] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 6097.424114] Key type lgssc registered [ 6097.762244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6099.998677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6100.004412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6101.146542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6102.285283] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6104.787255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6104.892151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6106.512993] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6108.005490] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6109.958027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6110.070220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6111.079416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6111.191695] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6115.297337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6115.300819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6115.304169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6116.322648] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6116.323487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6116.323649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6116.330161] Lustre: Skipped 1 previous similar message [ 6119.864078] LustreError: 425501:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6119.895215] Lustre: server umount lustre-OST0000 complete [ 6121.441618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6121.442326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6121.446757] Lustre: Skipped 1 previous similar message [ 6126.562096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6126.565699] Lustre: Skipped 2 previous similar messages [ 6127.119871] LustreError: 425703:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6127.123711] LustreError: 425703:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6127.160259] Lustre: server umount lustre-MDT0000 complete [ 6128.577185] LustreError: 423430:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357308 with bad export cookie 17542771559995562270 [ 6128.583153] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6128.586510] LustreError: 423430:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6128.681347] LustreError: 425906:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6128.683982] LustreError: 425906:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6128.737182] Lustre: server umount lustre-MDT0001 complete [ 6132.963763] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 6133.873251] Key type lgssc unregistered [ 6133.983322] LNet: 426437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6133.985356] LNetError: 426437:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6133.994275] LNet: Removed LNI 192.168.202.129@tcp [ 6134.256161] Key type .llcrypt unregistered [ 6134.257067] Key type ._llcrypt unregistered [ 6144.123374] Key type ._llcrypt registered [ 6144.124272] Key type .llcrypt registered [ 6144.157139] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 6144.534654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6144.541155] alg: No test for adler32 (adler32-zlib) [ 6145.385526] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 6145.462387] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 6147.040162] Key type lgssc registered [ 6147.595309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6150.238659] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6150.245867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6165.664239] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6170.784265] LustreError: 427784:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.129@tcp: failed processing log, type 4: rc = -110 [ 6197.408299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6199.822807] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6202.007961] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6205.286399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6205.566727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6205.613649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6207.267779] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6211.044161] LustreError: 427811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6211.054469] LustreError: 427811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6211.432290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6211.646307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6213.449545] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6215.728330] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6217.413445] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6221.281445] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6221.285740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6221.293379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6221.794238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6221.798682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6221.806883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6223.333984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6223.337530] Lustre: Skipped 1 previous similar message [ 6226.160168] LustreError: 430068:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 6226.215355] Lustre: server umount lustre-OST0000 complete [ 6228.449947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6228.459789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6228.463374] Lustre: Skipped 1 previous similar message [ 6233.569977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6233.573966] Lustre: Skipped 1 previous similar message [ 6242.272207] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6242.326950] LustreError: 430271:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6242.330315] LustreError: 430271:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6242.362267] Lustre: server umount lustre-MDT0000 complete [ 6243.809523] LustreError: 428713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6243.819097] LustreError: 428713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6244.121140] LustreError: 427792:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357424 with bad export cookie 9821718328966450812 [ 6244.128483] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6244.128699] LustreError: 427792:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6244.229520] LustreError: 430475:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6244.232685] LustreError: 430475:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6244.291232] Lustre: server umount lustre-MDT0001 complete [ 6248.811304] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 6250.140382] Key type lgssc unregistered [ 6250.271653] LNet: 431005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6250.276043] LNetError: 431005:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6250.284600] LNet: Removed LNI 192.168.202.129@tcp [ 6250.731318] Key type .llcrypt unregistered [ 6250.733383] Key type ._llcrypt unregistered [ 6261.600897] Key type ._llcrypt registered [ 6261.602851] Key type .llcrypt registered [ 6261.662976] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 6261.971045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6261.978451] alg: No test for adler32 (adler32-zlib) [ 6262.905819] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 6263.018847] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 6264.632161] Key type lgssc registered [ 6265.212928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6269.137494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6269.150110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6270.392454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6272.199342] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6276.410569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6276.647671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6278.396237] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6281.022608] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6282.738924] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6286.057617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6286.230966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6288.937176] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6291.538611] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6295.472762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6300.646525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6300.648636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6300.653805] Lustre: Skipped 2 previous similar messages [ 6300.657437] Lustre: Skipped 1 previous similar message [ 6301.667105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6301.670631] Lustre: Skipped 1 previous similar message [ 6302.952536] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.29@tcp (stopping) [ 6304.977757] LustreError: 434625:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6305.020416] Lustre: server umount lustre-MDT0000 complete [ 6306.087303] LustreError: 432362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.29@tcp arrived at 1763357486 with bad export cookie 5220498628897415660 [ 6306.095369] LustreError: 432362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6306.786738] LustreError: 432381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6306.795855] LustreError: 432381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6307.049737] LustreError: 432362:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357487 with bad export cookie 5220498628897415016 [ 6307.058600] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6311.907397] LustreError: 432381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6311.908496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6311.914824] LustreError: 432381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6311.923264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6312.930205] LustreError: 432381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6312.938562] LustreError: 432381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6313.140196] LustreError: 434828:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6313.142248] LustreError: 434828:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6313.198865] Lustre: server umount lustre-OST0000 complete [ 6315.960300] LustreError: 433095:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c3e11bdb x1849014294177536/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6315.976945] LustreError: 435184:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6315.980547] LustreError: 435184:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6316.036019] Lustre: server umount lustre-MDT0001 complete [ 6319.081047] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 6319.985902] Key type lgssc unregistered [ 6320.108312] LNet: 435714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6320.110291] LNetError: 435714:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6320.117391] LNet: Removed LNI 192.168.202.129@tcp [ 6320.404784] Key type .llcrypt unregistered [ 6320.405678] Key type ._llcrypt unregistered [ 6329.047976] Key type ._llcrypt registered [ 6329.048804] Key type .llcrypt registered [ 6329.085190] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 6329.416213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6329.439174] alg: No test for adler32 (adler32-zlib) [ 6330.293828] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 6330.381074] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 6331.960114] Key type lgssc registered [ 6332.293122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6334.765976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6334.771355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6335.916896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6337.063502] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6339.624023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6339.732341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6340.824676] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6342.360504] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6343.505641] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6345.625563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6345.724906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6346.791085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6347.294321] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6349.137517] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6354.915201] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6354.918539] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6354.919976] Lustre: Skipped 1 previous similar message [ 6355.679560] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.29@tcp (stopping) [ 6357.050967] LustreError: 439336:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6357.070552] Lustre: server umount lustre-MDT0001 complete [ 6360.034137] LustreError: 437813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6360.035933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6360.042908] LustreError: 437813:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6360.050289] Lustre: Skipped 1 previous similar message [ 6360.051273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6364.596128] LustreError: 439537:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6364.599816] LustreError: 439537:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6364.655483] Lustre: server umount lustre-OST0000 complete [ 6365.153046] LustreError: 437813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6365.925771] LustreError: 439738:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6365.927521] LustreError: 439738:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6365.975928] Lustre: server umount lustre-MDT0000 complete [ 6371.319044] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 6372.294862] Key type lgssc unregistered [ 6372.436783] LNet: 440422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6372.441120] LNetError: 440422:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6372.451591] LNet: Removed LNI 192.168.202.129@tcp [ 6372.888240] Key type .llcrypt unregistered [ 6372.890157] Key type ._llcrypt unregistered [ 6381.964407] Key type ._llcrypt registered [ 6381.965775] Key type .llcrypt registered [ 6382.005314] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 6388.324506] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 6388.704855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6388.712158] alg: No test for adler32 (adler32-zlib) [ 6389.566173] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 6389.656084] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 6391.232093] Key type lgssc registered [ 6391.523272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6394.380480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6396.257619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6398.947632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6401.088411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6404.633191] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 6407.585570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6410.480294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6410.498387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6410.502684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6411.593405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6411.611046] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6411.663121] Lustre: lustre-MDT0000: new disk, initializing [ 6411.694527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6411.704340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6413.445267] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6417.012845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6420.070427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6420.091351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6420.175496] Lustre: lustre-OST0000: new disk, initializing [ 6420.177681] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6420.179945] Lustre: Skipped 1 previous similar message [ 6420.199699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6421.938885] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6425.574905] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6426.113262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6426.115706] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6426.122918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6435.068664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6437.659591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6437.698712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6437.757972] Lustre: 444366:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6437.812065] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6437.864056] Lustre: lustre-MDT0001: new disk, initializing [ 6437.941415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6437.966061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6437.970922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6439.779337] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6444.090906] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6444.210296] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6445.675365] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6447.833175] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [ 6452.196978] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6452.201279] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6452.209375] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6452.214077] Lustre: Skipped 1 previous similar message [ 6454.504521] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.29@tcp (stopping) [ 6456.208320] LustreError: 447802:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6456.240978] Lustre: server umount lustre-MDT0001 complete [ 6462.434311] LustreError: 445977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6462.437795] LustreError: 445977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6467.556408] LustreError: 444402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6467.567071] LustreError: 444402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6472.161401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6472.165823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6472.172737] Lustre: Skipped 1 previous similar message [ 6472.175915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6472.674716] LustreError: 444377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6475.380908] LustreError: 448075:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6475.385510] LustreError: 448075:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6475.451505] Lustre: server umount lustre-OST0000 complete [ 6483.538048] LustreError: 448276:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6483.541635] LustreError: 448276:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6483.609688] Lustre: server umount lustre-MDT0000 complete [ 6489.952073] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 6491.397783] Key type lgssc unregistered [ 6491.533507] LNet: 448960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6491.537305] LNetError: 448960:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6491.547598] LNet: Removed LNI 192.168.202.129@tcp [ 6492.001225] Key type .llcrypt unregistered [ 6492.002851] Key type ._llcrypt unregistered [ 6508.910976] Key type ._llcrypt registered [ 6508.912028] Key type .llcrypt registered [ 6508.947586] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 6509.038218] Key type .llcrypt unregistered [ 6509.039127] Key type ._llcrypt unregistered [ 6515.135800] Key type ._llcrypt registered [ 6515.136676] Key type .llcrypt registered [ 6515.169182] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 6515.436918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6515.481268] alg: No test for adler32 (adler32-zlib) [ 6516.324290] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 6516.397626] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 6517.968102] Key type lgssc registered [ 6518.279660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6520.549713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6522.982706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6525.821659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6529.004936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6529.034093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6529.043128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6530.150035] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6530.168868] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6530.221431] Lustre: lustre-MDT0000: new disk, initializing [ 6530.254906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6530.267316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6531.985252] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6536.822918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6536.843090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6536.872799] Lustre: 452281:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6536.893097] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6536.894640] Lustre: Skipped 1 previous similar message [ 6536.929482] Lustre: lustre-MDT0001: new disk, initializing [ 6536.959816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6536.969949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6536.972897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6538.073726] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6540.879680] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6541.942653] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6543.978215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6543.998047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6544.089806] Lustre: lustre-OST0000: new disk, initializing [ 6544.092995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6544.118669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6545.837161] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6547.384820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6547.388305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6547.421203] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6549.558196] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6552.548575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6552.549120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6552.549802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6552.553297] Lustre: Skipped 2 previous similar messages [ 6557.665834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6557.669874] Lustre: Skipped 4 previous similar messages [ 6558.281437] LustreError: 454723:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6558.314326] Lustre: server umount lustre-MDT0000 complete [ 6559.829758] LustreError: 452272:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357740 with bad export cookie 7949760452367905159 [ 6559.835457] LustreError: 452272:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6559.836107] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6559.916697] LustreError: 454924:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6559.919674] LustreError: 454924:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6559.974623] Lustre: server umount lustre-MDT0001 complete [ 6571.574455] LustreError: 455125:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6571.578302] LustreError: 455125:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6571.631800] Lustre: server umount lustre-OST0000 complete [ 6576.057629] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:35:54 (1763357754) [ 6576.506576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6579.667128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6579.907479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6581.752169] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6585.314041] LustreError: 455952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6585.323021] LustreError: 455952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6585.823727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6586.019104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6587.754703] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6590.465180] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6592.272570] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6595.548610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6595.580137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6595.656300] Lustre: Found index 0 for lustre-OST0000, updating log [ 6595.743586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6598.025730] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6599.677083] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6604.971049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6607.329038] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6607.331621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6607.338042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6612.787405] LustreError: 458207:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6612.793402] LustreError: 458207:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6612.869666] Lustre: server umount lustre-OST0000 complete [ 6614.424460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6617.058512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6617.061037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6617.064367] Lustre: Skipped 1 previous similar message [ 6617.066420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6617.068804] Lustre: Skipped 3 previous similar messages [ 6619.624943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6624.748118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6624.758997] Lustre: Skipped 3 previous similar messages [ 6630.373885] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6630.452657] LustreError: 458462:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6630.458941] LustreError: 458462:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6630.564521] Lustre: server umount lustre-MDT0000 complete [ 6633.579626] LustreError: 455933:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357813 with bad export cookie 7949760452367906538 [ 6633.582922] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6633.585913] LustreError: 455933:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6633.915135] Lustre: server umount lustre-MDT0001 complete [ 6639.967784] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 00:36:58 (1763357818) [ 6640.433856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6640.831557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6641.182161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6641.517245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6641.861635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6642.193753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6642.509658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6642.844619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6643.173891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6643.486508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6643.824814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6644.169608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6644.511458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6644.829038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6645.168059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6645.503605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6645.824458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6646.160282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6646.495879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6646.876045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6647.208516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6647.556147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6647.897403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6648.247988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6648.576527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6648.902080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6649.225426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6649.549404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6649.893289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6650.233624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6650.583190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6650.928418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6651.277618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6651.616738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6651.959073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6652.293623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6652.615300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6652.947576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6653.293771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6653.642147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6653.995671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6654.333080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6654.640888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6654.960602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6655.292638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6655.605843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6655.934501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6656.242397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6656.561375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6656.884525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6657.199863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6659.619377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6659.746934] LustreError: 462046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6659.769831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6660.985993] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6663.684470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6665.060767] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6665.888322] LNet: 1 local NIs in recovery (showing 1): 192.168.202.129@tcp [ 6666.791554] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6668.038723] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6670.366522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6670.397018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6670.487941] Lustre: Found index 0 for lustre-OST0000, updating log [ 6671.590510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6672.501454] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6674.315256] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6676.960731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6676.962803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6676.966645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6676.968389] Lustre: Skipped 2 previous similar messages [ 6681.074375] LustreError: 464351:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6681.077580] LustreError: 464351:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6681.116788] Lustre: server umount lustre-OST0000 complete [ 6688.530466] Lustre: server umount lustre-MDT0000 complete [ 6688.545311] LustreError: 462046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6688.551849] LustreError: 462046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6689.798105] LustreError: 462027:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357870 with bad export cookie 7949760452367907609 [ 6689.801084] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6689.802317] LustreError: 462027:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6689.922174] Lustre: server umount lustre-MDT0001 complete [ 6690.913589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6695.997490] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 6698.955087] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 6702.078653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6704.113223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6706.090218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6708.071890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6710.199212] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 00:38:08 (1763357888) [ 6726.304131] LNet: 1 local NIs in recovery (showing 1): 192.168.202.129@tcp [ 6731.306232] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 6732.373736] Key type lgssc unregistered [ 6732.501550] LNet: 468334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6732.504864] LNetError: 468334:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6862.572692] LNet: Removed LNI 192.168.202.129@tcp [ 6863.112224] Key type .llcrypt unregistered [ 6863.114249] Key type ._llcrypt unregistered [ 6871.714938] Key type ._llcrypt registered [ 6871.716793] Key type .llcrypt registered [ 6871.785534] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 6872.154790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6872.163268] alg: No test for adler32 (adler32-zlib) [ 6873.059106] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 6873.171537] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 6874.776230] Key type lgssc registered [ 6875.433724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6877.544528] LNet: Added LNI 192.168.202.129@tcp100 [8/256/0/180] [ 6877.904302] LNet: Added LNI 192.168.202.129@tcp101 [8/256/0/180] [ 6878.233448] LNet: Added LNI 192.168.202.129@tcp102 [8/256/0/180] [ 6878.653360] LNet: Added LNI 192.168.202.129@tcp103 [8/256/0/180] [ 6878.981887] LNet: Added LNI 192.168.202.129@tcp104 [8/256/0/180] [ 6879.285254] LNet: Added LNI 192.168.202.129@tcp105 [8/256/0/180] [ 6879.629417] LNet: Added LNI 192.168.202.129@tcp106 [8/256/0/180] [ 6879.975217] LNet: Added LNI 192.168.202.129@tcp107 [8/256/0/180] [ 6880.349544] LNet: Added LNI 192.168.202.129@tcp108 [8/256/0/180] [ 6880.692960] LNet: Added LNI 192.168.202.129@tcp109 [8/256/0/180] [ 6881.055049] LNet: Added LNI 192.168.202.129@tcp110 [8/256/0/180] [ 6881.387802] LNet: Added LNI 192.168.202.129@tcp111 [8/256/0/180] [ 6881.674650] LNet: Added LNI 192.168.202.129@tcp112 [8/256/0/180] [ 6881.936990] LNet: Added LNI 192.168.202.129@tcp113 [8/256/0/180] [ 6882.197260] LNet: Added LNI 192.168.202.129@tcp114 [8/256/0/180] [ 6882.464816] LNet: Added LNI 192.168.202.129@tcp115 [8/256/0/180] [ 6882.740551] LNet: Added LNI 192.168.202.129@tcp116 [8/256/0/180] [ 6883.005301] LNet: Added LNI 192.168.202.129@tcp117 [8/256/0/180] [ 6883.293666] LNet: Added LNI 192.168.202.129@tcp118 [8/256/0/180] [ 6883.564529] LNet: Added LNI 192.168.202.129@tcp119 [8/256/0/180] [ 6883.835686] LNet: Added LNI 192.168.202.129@tcp120 [8/256/0/180] [ 6884.100342] LNet: Added LNI 192.168.202.129@tcp121 [8/256/0/180] [ 6884.374379] LNet: Added LNI 192.168.202.129@tcp122 [8/256/0/180] [ 6884.645017] LNet: Added LNI 192.168.202.129@tcp123 [8/256/0/180] [ 6884.925349] LNet: Added LNI 192.168.202.129@tcp124 [8/256/0/180] [ 6885.192692] LNet: Added LNI 192.168.202.129@tcp125 [8/256/0/180] [ 6885.464253] LNet: Added LNI 192.168.202.129@tcp126 [8/256/0/180] [ 6885.738394] LNet: Added LNI 192.168.202.129@tcp127 [8/256/0/180] [ 6886.009773] LNet: Added LNI 192.168.202.129@tcp128 [8/256/0/180] [ 6886.299971] LNet: Added LNI 192.168.202.129@tcp129 [8/256/0/180] [ 6886.565652] LNet: Added LNI 192.168.202.129@tcp130 [8/256/0/180] [ 6886.850125] LNet: Added LNI 192.168.202.129@tcp131 [8/256/0/180] [ 6887.110663] LNet: Added LNI 192.168.202.129@tcp132 [8/256/0/180] [ 6887.392833] LNet: Added LNI 192.168.202.129@tcp133 [8/256/0/180] [ 6887.680280] LNet: Added LNI 192.168.202.129@tcp134 [8/256/0/180] [ 6887.960498] LNet: Added LNI 192.168.202.129@tcp135 [8/256/0/180] [ 6890.040392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6890.060529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6890.064801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6891.101259] Lustre: 471323:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.129@tcp: can't add NID 192.168.202.129@tcp130: rc = -75 [ 6891.129441] LNetError: 471347:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.129 [ 6891.132625] LNetError: Protocol error connecting to 192.168.202.129@tcp117 at host 192.168.202.129:988: is it running a compatible version of Lustre? [ 6891.172651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6891.182255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6891.216846] Lustre: lustre-MDT0000: new disk, initializing [ 6891.244724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6891.251184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6892.130127] LNetError: 468784:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.129 [ 6892.132785] LNetError: Protocol error connecting to 192.168.202.129@tcp101 at host 192.168.202.129:988: is it running a compatible version of Lustre? [ 6892.362896] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6896.287641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6896.314820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6896.355974] Lustre: 471341:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6896.420403] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6896.420574] Lustre: 472130:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.129@tcp: can't add NID 192.168.202.129@tcp130: rc = -75 [ 6896.421884] Lustre: Skipped 1 previous similar message [ 6896.426937] Lustre: 472130:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6896.436456] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6896.464249] Lustre: lustre-MDT0001: new disk, initializing [ 6896.516298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6896.531177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6896.534248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6897.774499] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6900.798279] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6901.901382] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6904.834289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6904.876141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6905.018226] Lustre: lustre-OST0000: new disk, initializing [ 6905.019965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6905.055730] Lustre: 473098:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.129@tcp: can't add NID 192.168.202.129@tcp130: rc = -75 [ 6905.058511] Lustre: 473098:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [ 6905.073232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6905.566723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6905.570890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6905.620394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6906.900900] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 6909.812673] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6916.068274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6916.071597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6916.075652] LustreError: Skipped 1 previous similar message [ 6916.084396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6916.090535] Lustre: Skipped 1 previous similar message [ 6917.999179] LustreError: 473936:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6918.042504] Lustre: server umount lustre-OST0000 complete [ 6921.186390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6921.186876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6921.189207] Lustre: Skipped 2 previous similar messages [ 6924.385274] LNetError: 468785:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.129 [ 6924.390111] LNetError: Protocol error connecting to 192.168.202.129@tcp101 at host 192.168.202.129:988: is it running a compatible version of Lustre? [ 6925.693468] LustreError: 474137:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6925.695124] LustreError: 474137:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6925.722827] Lustre: server umount lustre-MDT0000 complete [ 6926.432782] LNetError: 471345:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.129 [ 6926.436652] LNetError: Protocol error connecting to 192.168.202.129@tcp117 at host 192.168.202.129:988: is it running a compatible version of Lustre? [ 6926.870235] LustreError: 472128:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763358107 with bad export cookie 5747640231699732362 [ 6926.873341] LustreError: 472128:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6926.873734] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6926.939257] LustreError: 474338:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6926.940769] LustreError: 474338:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6926.986110] Lustre: server umount lustre-MDT0001 complete [ 6930.179191] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 6931.080844] Key type lgssc unregistered [ 6931.225527] LNet: 474876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6931.227686] LNetError: 474876:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6931.229510] LNet: Removed LNI 192.168.202.129@tcp [ 6932.256165] LNet: Removed LNI 192.168.202.129@tcp100 [ 6932.257642] LNetError: 474876:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6932.259796] LNetError: 474876:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6932.262161] LNet: Removed LNI 192.168.202.129@tcp101 [ 6933.280154] LNet: Removed LNI 192.168.202.129@tcp102 [ 6933.281761] LNetError: 474876:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6933.284161] LNetError: 474876:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6933.286809] LNet: Removed LNI 192.168.202.129@tcp103 [ 6934.304188] LNet: Removed LNI 192.168.202.129@tcp104 [ 6935.328129] LNet: Removed LNI 192.168.202.129@tcp105 [ 6935.329562] LNetError: 474876:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6935.331667] LNetError: 474876:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 6935.333915] LNet: Removed LNI 192.168.202.129@tcp106 [ 6936.352153] LNet: Removed LNI 192.168.202.129@tcp107 [ 6937.376108] LNet: Removed LNI 192.168.202.129@tcp108 [ 6938.400156] LNet: Removed LNI 192.168.202.129@tcp109 [ 6939.424104] LNet: Removed LNI 192.168.202.129@tcp110 [ 6939.425173] LNetError: 474876:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6939.426855] LNetError: 474876:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 6939.428665] LNet: Removed LNI 192.168.202.129@tcp111 [ 6940.448202] LNet: Removed LNI 192.168.202.129@tcp112 [ 6941.472109] LNet: Removed LNI 192.168.202.129@tcp113 [ 6942.496159] LNet: Removed LNI 192.168.202.129@tcp114 [ 6943.520086] LNet: Removed LNI 192.168.202.129@tcp115 [ 6944.544131] LNet: Removed LNI 192.168.202.129@tcp116 [ 6945.568129] LNet: Removed LNI 192.168.202.129@tcp117 [ 6945.569314] LNet: Removed LNI 192.168.202.129@tcp118 [ 6945.570351] LNet: Removed LNI 192.168.202.129@tcp119 [ 6945.571467] LNet: Removed LNI 192.168.202.129@tcp120 [ 6945.572465] LNet: Removed LNI 192.168.202.129@tcp121 [ 6945.573417] LNet: Removed LNI 192.168.202.129@tcp122 [ 6945.574356] LNet: Removed LNI 192.168.202.129@tcp123 [ 6945.575282] LNet: Removed LNI 192.168.202.129@tcp124 [ 6945.576228] LNet: Removed LNI 192.168.202.129@tcp125 [ 6945.577159] LNet: Removed LNI 192.168.202.129@tcp126 [ 6945.578087] LNet: Removed LNI 192.168.202.129@tcp127 [ 6945.578988] LNet: Removed LNI 192.168.202.129@tcp128 [ 6945.579915] LNet: Removed LNI 192.168.202.129@tcp129 [ 6945.580890] LNet: Removed LNI 192.168.202.129@tcp130 [ 6945.581845] LNet: Removed LNI 192.168.202.129@tcp131 [ 6945.582776] LNet: Removed LNI 192.168.202.129@tcp132 [ 6945.583738] LNet: Removed LNI 192.168.202.129@tcp133 [ 6945.584682] LNet: Removed LNI 192.168.202.129@tcp134 [ 6945.591496] LNet: Removed LNI 192.168.202.129@tcp135 [ 6945.828646] Key type .llcrypt unregistered [ 6945.830071] Key type ._llcrypt unregistered [ 6954.555463] Key type ._llcrypt registered [ 6954.556458] Key type .llcrypt registered [ 6954.594298] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 6961.807361] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 6962.146798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6962.195586] alg: No test for adler32 (adler32-zlib) [ 6963.047199] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 6963.120327] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 6964.696111] Key type lgssc registered [ 6965.040040] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6967.596333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6969.560479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6972.405171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6975.258671] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6977.808234] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 00:42:36 (1763358156) [ 6983.428330] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 6984.548888] Key type lgssc unregistered [ 6984.684500] LNet: 478891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6984.688613] LNetError: 478891:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6984.696501] LNet: Removed LNI 192.168.202.129@tcp [ 6985.103814] Key type .llcrypt unregistered [ 6985.105996] Key type ._llcrypt unregistered [ 6992.951591] Key type ._llcrypt registered [ 6992.952796] Key type .llcrypt registered [ 6992.994302] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 6993.308336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6993.364818] alg: No test for adler32 (adler32-zlib) [ 6994.226852] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 6994.344915] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 6995.936287] Key type lgssc registered [ 6996.591801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7001.158517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7001.188503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7001.196991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7002.294300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7002.306839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7002.351947] Lustre: lustre-MDT0000: new disk, initializing [ 7002.375335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7002.382041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7003.999855] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7009.395273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7009.428717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7009.486675] Lustre: 480062:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7009.536209] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7009.539590] Lustre: Skipped 1 previous similar message [ 7009.587520] Lustre: lustre-MDT0001: new disk, initializing [ 7009.642907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7009.660986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7009.669524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7011.597421] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7015.716233] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7017.384380] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7019.894240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7019.925050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7020.030827] Lustre: lustre-OST0000: new disk, initializing [ 7020.032538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7020.063584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7021.311302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7021.314165] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7021.356182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7021.873066] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7024.745193] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7025.802464] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing lnet_if_list [ 7027.285336] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7027.295706] LNet: Added LNI 192.168.202.129@tcp10 [8/256/0/180] [ 7028.873466] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7028.885937] LNet: Added LNI 192.168.202.129@tcp11 [8/256/0/180] [ 7030.045623] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7030.054960] LNet: Added LNI 192.168.202.129@tcp12 [8/256/0/180] [ 7031.237925] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7031.247241] LNet: Added LNI 192.168.202.129@tcp13 [8/256/0/180] [ 7032.516788] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7032.526416] LNet: Added LNI 192.168.202.129@tcp14 [8/256/0/180] [ 7033.894257] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7033.905032] LNet: Added LNI 192.168.202.129@tcp15 [8/256/0/180] [ 7035.273639] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7035.283019] LNet: Added LNI 192.168.202.129@tcp16 [8/256/0/180] [ 7036.649018] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7036.658747] LNet: Added LNI 192.168.202.129@tcp17 [8/256/0/180] [ 7038.025433] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7038.034704] LNet: Added LNI 192.168.202.129@tcp18 [8/256/0/180] [ 7039.364613] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7039.372824] LNet: Added LNI 192.168.202.129@tcp19 [8/256/0/180] [ 7040.668345] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7040.678211] LNet: Added LNI 192.168.202.129@tcp20 [8/256/0/180] [ 7042.048195] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7042.057701] LNet: Added LNI 192.168.202.129@tcp21 [8/256/0/180] [ 7043.449492] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7043.458580] LNet: Added LNI 192.168.202.129@tcp22 [8/256/0/180] [ 7044.850188] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7044.858492] LNet: Added LNI 192.168.202.129@tcp23 [8/256/0/180] [ 7046.333838] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7046.343571] LNet: Added LNI 192.168.202.129@tcp24 [8/256/0/180] [ 7047.822808] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7047.831633] LNet: Added LNI 192.168.202.129@tcp25 [8/256/0/180] [ 7049.380795] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7049.392744] LNet: Added LNI 192.168.202.129@tcp26 [8/256/0/180] [ 7051.160814] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7051.169878] LNet: Added LNI 192.168.202.129@tcp27 [8/256/0/180] [ 7051.177385] LNetError: 479323:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.129 [ 7051.180146] LNetError: Protocol error connecting to 192.168.202.129@tcp26 at host 192.168.202.129:988: is it running a compatible version of Lustre? [ 7052.936955] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7052.947061] LNet: Added LNI 192.168.202.129@tcp28 [8/256/0/180] [ 7052.950712] LNetError: 479321:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.129 [ 7052.955185] LNetError: Protocol error connecting to 192.168.202.129@tcp26 at host 192.168.202.129:988: is it running a compatible version of Lustre? [ 7054.596781] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7054.606702] LNet: Added LNI 192.168.202.129@tcp29 [8/256/0/180] [ 7054.612791] LNetError: 479322:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.129 [ 7054.615936] LNetError: Protocol error connecting to 192.168.202.129@tcp28 at host 192.168.202.129:988: is it running a compatible version of Lustre? [ 7054.752100] LNet: 1 local NIs in recovery (showing 1): 192.168.202.129@tcp28 [ 7056.203376] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7056.211687] LNet: Added LNI 192.168.202.129@tcp30 [8/256/0/180] [ 7057.826917] LNetError: 479321:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.129 [ 7057.832724] LNetError: 479321:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [ 7057.837277] LNetError: Protocol error connecting to 192.168.202.129@tcp28 at host 192.168.202.129:988: is it running a compatible version of Lustre? [ 7057.842647] LNetError: Skipped 1 previous similar message [ 7058.189959] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7058.203990] LNet: Added LNI 192.168.202.129@tcp31 [8/256/0/180] [ 7060.065942] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7060.074555] LNet: Added LNI 192.168.202.129@tcp32 [8/256/0/180] [ 7061.924133] LNetError: 479321:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.129 [ 7061.929061] LNetError: 479321:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 6 previous similar messages [ 7061.933181] LNetError: Protocol error connecting to 192.168.202.129@tcp29 at host 192.168.202.129:988: is it running a compatible version of Lustre? [ 7061.938960] LNetError: Skipped 6 previous similar messages [ 7061.982883] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7061.995992] LNet: Added LNI 192.168.202.129@tcp33 [8/256/0/180] [ 7064.190338] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7064.198686] LNet: Added LNI 192.168.202.129@tcp34 [8/256/0/180] [ 7066.284562] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7066.296184] LNet: Added LNI 192.168.202.129@tcp35 [8/256/0/180] [ 7068.067899] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7068.076520] LNet: Added LNI 192.168.202.129@tcp36 [8/256/0/180] [ 7069.827674] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7069.835935] LNet: Added LNI 192.168.202.129@tcp37 [8/256/0/180] [ 7071.137957] LNetError: 479321:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.129 [ 7071.140423] LNetError: 479321:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 9 previous similar messages [ 7071.142216] LNetError: Protocol error connecting to 192.168.202.129@tcp34 at host 192.168.202.129:988: is it running a compatible version of Lustre? [ 7071.144531] LNetError: Skipped 9 previous similar messages [ 7071.646812] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7071.655268] LNet: Added LNI 192.168.202.129@tcp38 [8/256/0/180] [ 7073.482914] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7073.491801] LNet: Added LNI 192.168.202.129@tcp39 [8/256/0/180] [ 7075.553181] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7075.564828] LNet: Added LNI 192.168.202.129@tcp40 [8/256/0/180] [ 7078.133563] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7078.147246] LNet: Added LNI 192.168.202.129@tcp41 [8/256/0/180] [ 7080.440717] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7080.449635] LNet: Added LNI 192.168.202.129@tcp42 [8/256/0/180] [ 7082.385517] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7082.393483] LNet: Added LNI 192.168.202.129@tcp43 [8/256/0/180] [ 7084.393996] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7084.404915] LNet: Added LNI 192.168.202.129@tcp44 [8/256/0/180] [ 7086.406275] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7086.416257] LNet: Added LNI 192.168.202.129@tcp45 [8/256/0/180] [ 7087.523456] LNetError: 479323:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.129 [ 7087.526770] LNetError: 479323:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 31 previous similar messages [ 7087.529520] LNetError: Protocol error connecting to 192.168.202.129@tcp26 at host 192.168.202.129:988: is it running a compatible version of Lustre? [ 7087.532758] LNetError: Skipped 32 previous similar messages [ 7088.519041] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7088.527469] LNet: Added LNI 192.168.202.129@tcp46 [8/256/0/180] [ 7090.665283] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7090.674464] LNet: Added LNI 192.168.202.129@tcp47 [8/256/0/180] [ 7092.776913] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7092.787723] LNet: Added LNI 192.168.202.129@tcp48 [8/256/0/180] [ 7094.920654] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7094.929662] LNet: Added LNI 192.168.202.129@tcp49 [8/256/0/180] [ 7097.472584] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7097.489242] LNet: Added LNI 192.168.202.129@tcp50 [8/256/0/180] [ 7108.583206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7108.583252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7108.591459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7108.594289] LustreError: Skipped 1 previous similar message [ 7108.600159] Lustre: Skipped 1 previous similar message [ 7112.500139] LustreError: 504242:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7112.560461] Lustre: server umount lustre-OST0000 complete [ 7115.168241] LNet: 5 local NIs in recovery (showing 5): 192.168.202.129@tcp38, 192.168.202.129@tcp44, 192.168.202.129@tcp34, 192.168.202.129@tcp33, 192.168.202.129@tcp29 [ 7117.281517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7117.285781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7117.292351] Lustre: Skipped 1 previous similar message [ 7117.295546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7118.817203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7118.817753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7120.527563] LustreError: 504443:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7120.532027] LustreError: 504443:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7120.571059] Lustre: server umount lustre-MDT0000 complete [ 7122.336237] LustreError: 480052:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763358302 with bad export cookie 6044910306298173449 [ 7122.340650] LNetError: 479323:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.129 [ 7122.345487] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7122.346511] LustreError: 480052:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7122.351943] LNetError: 479323:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 39 previous similar messages [ 7122.352082] LNetError: Protocol error connecting to 192.168.202.129@tcp41 at host 192.168.202.129:988: is it running a compatible version of Lustre? [ 7122.371355] LNetError: Skipped 38 previous similar messages [ 7122.435421] LustreError: 504645:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7122.439181] LustreError: 504645:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7122.494419] Lustre: server umount lustre-MDT0001 complete [ 7126.909734] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 7128.349820] Key type lgssc unregistered [ 7128.557825] LNet: 505185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7128.563201] LNetError: 505185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7128.567385] LNet: Removed LNI 192.168.202.129@tcp [ 7129.632098] LNet: Removed LNI 192.168.202.129@tcp10 [ 7129.633478] LNetError: 505185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7129.635098] LNetError: 505185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7129.636879] LNet: Removed LNI 192.168.202.129@tcp11 [ 7130.656214] LNet: Removed LNI 192.168.202.129@tcp12 [ 7130.658614] LNetError: 505185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7130.662696] LNetError: 505185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7130.667245] LNet: Removed LNI 192.168.202.129@tcp13 [ 7131.680213] LNet: Removed LNI 192.168.202.129@tcp14 [ 7132.704178] LNet: Removed LNI 192.168.202.129@tcp15 [ 7132.706542] LNetError: 505185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7132.710621] LNetError: 505185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7132.715393] LNet: Removed LNI 192.168.202.129@tcp16 [ 7133.728205] LNet: Removed LNI 192.168.202.129@tcp17 [ 7134.752112] LNet: Removed LNI 192.168.202.129@tcp18 [ 7135.776110] LNet: Removed LNI 192.168.202.129@tcp19 [ 7136.800195] LNet: Removed LNI 192.168.202.129@tcp20 [ 7136.804657] LNetError: 505185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7136.809954] LNetError: 505185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7136.814811] LNet: Removed LNI 192.168.202.129@tcp21 [ 7137.824239] LNet: Removed LNI 192.168.202.129@tcp22 [ 7138.848227] LNet: Removed LNI 192.168.202.129@tcp23 [ 7139.872110] LNet: Removed LNI 192.168.202.129@tcp24 [ 7140.896235] LNet: Removed LNI 192.168.202.129@tcp25 [ 7141.920180] LNet: Removed LNI 192.168.202.129@tcp26 [ 7142.944193] LNet: Removed LNI 192.168.202.129@tcp27 [ 7143.968190] LNet: Removed LNI 192.168.202.129@tcp28 [ 7144.992147] LNet: Removed LNI 192.168.202.129@tcp29 [ 7144.993300] LNetError: 505185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7144.995034] LNetError: 505185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7144.996821] LNet: Removed LNI 192.168.202.129@tcp30 [ 7146.016213] LNet: Removed LNI 192.168.202.129@tcp31 [ 7147.040181] LNet: Removed LNI 192.168.202.129@tcp32 [ 7147.042643] LNet: Removed LNI 192.168.202.129@tcp33 [ 7147.044948] LNet: Removed LNI 192.168.202.129@tcp34 [ 7148.064168] LNet: Removed LNI 192.168.202.129@tcp35 [ 7149.088292] LNet: Removed LNI 192.168.202.129@tcp36 [ 7150.112269] LNet: Removed LNI 192.168.202.129@tcp37 [ 7150.113790] LNet: Removed LNI 192.168.202.129@tcp38 [ 7151.136194] LNet: Removed LNI 192.168.202.129@tcp39 [ 7152.160193] LNet: Removed LNI 192.168.202.129@tcp40 [ 7153.184176] LNet: Removed LNI 192.168.202.129@tcp41 [ 7154.208198] LNet: Removed LNI 192.168.202.129@tcp42 [ 7155.232145] LNet: Removed LNI 192.168.202.129@tcp43 [ 7156.256251] LNet: Removed LNI 192.168.202.129@tcp44 [ 7157.280190] LNet: Removed LNI 192.168.202.129@tcp45 [ 7158.304210] LNet: Removed LNI 192.168.202.129@tcp46 [ 7159.328198] LNet: Removed LNI 192.168.202.129@tcp47 [ 7160.352137] LNet: Removed LNI 192.168.202.129@tcp48 [ 7160.353341] LNet: Removed LNI 192.168.202.129@tcp49 [ 7160.359366] LNet: Removed LNI 192.168.202.129@tcp50 [ 7160.597651] Key type .llcrypt unregistered [ 7160.598741] Key type ._llcrypt unregistered [ 7169.461951] Key type ._llcrypt registered [ 7169.464205] Key type .llcrypt registered [ 7169.528675] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 7169.950718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7169.959707] alg: No test for adler32 (adler32-zlib) [ 7170.803334] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 7170.879984] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 7172.456187] Key type lgssc registered [ 7173.051776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7176.861134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7176.873765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7178.062652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7179.717508] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7183.879990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7184.051183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7185.729755] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7188.303592] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7189.941952] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7193.251475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7193.447265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7195.707455] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7197.270402] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7210.838234] loop0: detected capacity change from 0 to 204800000 [ 7210.889827] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7210.892551] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7210.895422] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7210.916082] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7212.654678] loop0: detected capacity change from 0 to 204800000 [ 7212.705243] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7212.708399] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7212.712159] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7212.756781] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7214.258609] loop0: detected capacity change from 0 to 204800000 [ 7215.980339] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7215.996971] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7216.036875] Lustre: fs15246-OST0000: new disk, initializing [ 7216.038717] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7216.053617] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7217.854346] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7221.517778] loop1: detected capacity change from 0 to 204800000 [ 7223.432655] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7223.452180] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7223.479483] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7223.498667] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7223.542209] Lustre: fs15246-MDT0000: new disk, initializing [ 7223.588211] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7223.597927] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7223.632659] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7224.672899] Lustre: 510805:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000f12d6775 x1849015246290688/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763358421 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7224.844967] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7229.729965] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7229.734767] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7229.744826] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7234.027175] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7234.036656] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7238.985194] LustreError: 511931:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 7239.025267] Lustre: server umount fs15246-MDT0000 complete [ 7239.144374] LustreError: 506406:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7239.149677] LustreError: 506406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7242.101159] LustreError: 512434:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 7242.103881] LustreError: 512434:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7242.124409] Lustre: server umount fs15246-OST0000 complete [ 7248.865246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7248.869079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7248.875947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7254.500780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7254.504339] Lustre: Skipped 3 previous similar messages [ 7259.619279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7259.621258] Lustre: Skipped 1 previous similar message [ 7260.128072] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7260.148163] LustreError: 512935:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7260.149872] LustreError: 512935:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7260.183768] Lustre: server umount lustre-OST0000 complete [ 7264.738557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7264.740258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7264.748152] Lustre: Skipped 2 previous similar messages [ 7264.754300] Lustre: Skipped 1 previous similar message [ 7267.982711] LustreError: 513140:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7267.987864] LustreError: 513140:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7268.022164] Lustre: server umount lustre-MDT0000 complete [ 7269.793176] LustreError: 506390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763358450 with bad export cookie 11681180251309753843 [ 7269.799639] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7269.800526] LustreError: 506390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7269.937220] Lustre: server umount lustre-MDT0001 complete [ 7274.185292] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 7275.579583] Key type lgssc unregistered [ 7275.735649] LNet: 513872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7275.740230] LNetError: 513872:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7275.754542] LNet: Removed LNI 192.168.202.129@tcp [ 7276.222130] Key type .llcrypt unregistered [ 7276.224091] Key type ._llcrypt unregistered [ 7284.115313] Key type ._llcrypt registered [ 7284.116319] Key type .llcrypt registered [ 7284.163361] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 7284.519889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7284.538967] alg: No test for adler32 (adler32-zlib) [ 7285.383741] Lustre: Lustre: Build Version: 2.16.61_41_g6a6a76f [ 7285.464739] LNet: Added LNI 192.168.202.129@tcp [8/256/0/180] [ 7287.048174] Key type lgssc registered [ 7287.629697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7289.802645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7291.175723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7292.595888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7293.946143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7299.993889] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 7303.939262] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 7308.114864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7311.185412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7314.078611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7317.166462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7320.194102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7320.223127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7320.229865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7321.320334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7321.330829] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7321.372101] Lustre: lustre-MDT0000: new disk, initializing [ 7321.397187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7321.404414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7322.900053] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7328.376883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7328.419202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7328.473052] Lustre: 518274:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7328.510881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7328.514237] Lustre: Skipped 1 previous similar message [ 7328.559221] Lustre: lustre-MDT0001: new disk, initializing [ 7328.601361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7328.618050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7328.622290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7330.214395] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7334.068286] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7335.655573] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7338.854379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7338.896207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7339.046601] Lustre: lustre-OST0000: new disk, initializing [ 7339.050713] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7339.104166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7341.597295] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_default_debug -1 all [ 7345.461698] Lustre: DEBUG MARKER: oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7347.148166] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7347.722846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7347.729237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7347.776856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7348.293409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 7350.010211] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7350.124950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7352.802587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7352.803315] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7352.804847] LustreError: Skipped 1 previous similar message [ 7352.808959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7352.811524] Lustre: Skipped 1 previous similar message [ 7357.107157] LustreError: 521222:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7357.165529] Lustre: server umount lustre-OST0000 complete [ 7359.457578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7359.462865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7359.470995] Lustre: Skipped 1 previous similar message [ 7359.474640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7363.041977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7363.042895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7368.161345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7368.162975] Lustre: Skipped 2 previous similar messages [ 7373.280196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7373.282452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7373.292655] Lustre: Skipped 1 previous similar message [ 7373.343615] LustreError: 521423:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7373.347101] LustreError: 521423:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7373.381979] Lustre: server umount lustre-MDT0000 complete [ 7375.092236] LustreError: 519054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763358555 with bad export cookie 11021980074182626952 [ 7375.098292] LustreError: MGC192.168.202.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7375.098779] LustreError: 519054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7375.204619] LustreError: 521627:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7375.206826] LustreError: 521627:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7375.256431] Lustre: server umount lustre-MDT0001 complete [ 7385.661405] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing set_hostid [ 7389.860280] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing load_modules_local [ 7394.318259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7397.580810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7400.619519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7403.812582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7404.539398] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7187 sec ========== 00:49:43 (1763358583) [ 7405.357474] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:49:43 (1763358583) === [ 7406.873345] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:49:45 (1763358585) === [ 7424.934515] Lustre: DEBUG MARKER: oleg229-server.virtnet: executing unload_modules_local [ 7426.436248] Key type lgssc unregistered [ 7426.605642] LNet: 526478:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7426.609932] LNetError: 526478:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7426.622600] LNet: Removed LNI 192.168.202.129@tcp [ 7427.063813] Key type .llcrypt unregistered [ 7427.065723] Key type ._llcrypt unregistered