[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 616401644 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002017] kvm-guest: setup PV IPIs [ 0.005000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.005000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.005026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.006012] pid_max: default: 32768 minimum: 301 [ 0.007149] LSM: Security Framework initializing [ 0.008050] Yama: becoming mindful. [ 0.009031] SELinux: Initializing. [ 0.010089] *** VALIDATE selinux *** [ 0.019674] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024476] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025178] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027143] *** VALIDATE tmpfs *** [ 0.029199] *** VALIDATE proc *** [ 0.030235] *** VALIDATE cgroup *** [ 0.031009] *** VALIDATE cgroup2 *** [ 0.032295] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034168] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036034] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.040658] debug: unmapping init [mem 0xffffffffb6459000-0xffffffffb6460fff] [ 0.042184] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043754] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044026] ... version: 2 [ 0.045013] ... bit width: 48 [ 0.046010] ... generic registers: 4 [ 0.047009] ... value mask: 0000ffffffffffff [ 0.048010] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.050010] ... event mask: 000000070000000f [ 0.051326] rcu: Hierarchical SRCU implementation. [ 0.053603] smp: Bringing up secondary CPUs ... [ 0.054573] x86: Booting SMP configuration: [ 0.055026] .... node #0, CPUs: #1 #2 #3 [ 0.067654] smp: Brought up 1 node, 4 CPUs [ 0.069010] smpboot: Max logical packages: 1 [ 0.070012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143029] node 0 deferred pages initialised in 69ms [ 0.147203] devtmpfs: initialized [ 0.148299] x86/mm: Memory block size: 128MB [ 0.150755] gcov: version magic: 0x41383552 [ 0.153000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158102] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161385] pinctrl core: initialized pinctrl subsystem [ 0.162144] [ 0.162663] ************************************************************* [ 0.164011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166010] ** ** [ 0.168010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170010] ** ** [ 0.172012] ** This means that this kernel is built to expose internal ** [ 0.173000] ** IOMMU data structures, which may compromise security on ** [ 0.175015] ** your system. ** [ 0.177010] ** ** [ 0.180012] ** If you see this message and you are not debugging the ** [ 0.182010] ** kernel, report this immediately to your vendor! ** [ 0.184009] ** ** [ 0.185010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186011] ************************************************************* [ 0.188850] NET: Registered protocol family 16 [ 0.190686] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.197013] cpuidle: using governor menu [ 0.200000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201862] PCI: Using configuration type 1 for base access [ 0.203138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214000] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.217234] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220334] cryptd: max_cpu_qlen set to 1000 [ 0.225807] ACPI: Added _OSI(Module Device) [ 0.226017] ACPI: Added _OSI(Processor Device) [ 0.227040] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.229019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.235340] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.255887] ACPI: Interpreter enabled [ 0.256117] ACPI: PM: (supports S0 S3 S4 S5) [ 0.258036] ACPI: Using IOAPIC for interrupt routing [ 0.261127] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.265539] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.276816] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.278037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.279000] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.281089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.285172] acpiphp: Slot [2] registered [ 0.288178] acpiphp: Slot [3] registered [ 0.290092] acpiphp: Slot [4] registered [ 0.291084] acpiphp: Slot [5] registered [ 0.292000] acpiphp: Slot [6] registered [ 0.292000] acpiphp: Slot [7] registered [ 0.294161] acpiphp: Slot [8] registered [ 0.295094] acpiphp: Slot [9] registered [ 0.297093] acpiphp: Slot [10] registered [ 0.298090] acpiphp: Slot [11] registered [ 0.299000] acpiphp: Slot [12] registered [ 0.299000] acpiphp: Slot [13] registered [ 0.300100] acpiphp: Slot [14] registered [ 0.302089] acpiphp: Slot [15] registered [ 0.303085] acpiphp: Slot [16] registered [ 0.304078] acpiphp: Slot [17] registered [ 0.306076] acpiphp: Slot [18] registered [ 0.307102] acpiphp: Slot [19] registered [ 0.308068] acpiphp: Slot [20] registered [ 0.309059] acpiphp: Slot [21] registered [ 0.310116] acpiphp: Slot [22] registered [ 0.312056] acpiphp: Slot [23] registered [ 0.313075] acpiphp: Slot [24] registered [ 0.314307] acpiphp: Slot [25] registered [ 0.315075] acpiphp: Slot [26] registered [ 0.317076] acpiphp: Slot [27] registered [ 0.318072] acpiphp: Slot [28] registered [ 0.320108] acpiphp: Slot [29] registered [ 0.321096] acpiphp: Slot [30] registered [ 0.322100] acpiphp: Slot [31] registered [ 0.324078] PCI host bridge to bus 0000:00 [ 0.325015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.327017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.329015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.332019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.335018] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.339027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.340171] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.348714] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.352000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.358016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.366051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.369019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.375029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.377014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.382496] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.385389] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.388029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.389588] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.394017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.408014] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.412767] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.418221] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.427026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.441026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.459022] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.471103] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.482023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.493020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.516021] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.530367] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.538022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.548025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.571021] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.596021] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.601019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.614034] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.642020] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.653000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.662023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.668017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.718022] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.732000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.743030] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.751000] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.771016] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.775000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.777309] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.779359] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.781372] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.783219] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.789107] iommu: Default domain type: Passthrough [ 0.791532] SCSI subsystem initialized [ 0.792162] ACPI: bus type USB registered [ 0.793237] usbcore: registered new interface driver usbfs [ 0.796098] usbcore: registered new interface driver hub [ 0.798149] usbcore: registered new device driver usb [ 0.800178] pps_core: LinuxPPS API ver. 1 registered [ 0.801007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.804091] PTP clock support registered [ 0.806654] EDAC MC: Ver: 3.0.0 [ 0.808528] PCI: Using ACPI for IRQ routing [ 0.811736] NetLabel: Initializing [ 0.812009] NetLabel: domain hash size = 128 [ 0.814013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.816104] NetLabel: unlabeled traffic allowed by default [ 0.818358] vgaarb: loaded [ 0.820013] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.822032] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.830799] clocksource: Switched to clocksource kvm-clock [ 0.976533] VFS: Disk quotas dquot_6.6.0 [ 0.977901] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.980276] *** VALIDATE ramfs *** [ 0.981570] *** VALIDATE hugetlbfs *** [ 0.983104] pnp: PnP ACPI init [ 0.985869] pnp: PnP ACPI: found 6 devices [ 1.001266] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.004224] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.006203] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.008612] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.010976] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.013329] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.016509] NET: Registered protocol family 2 [ 1.020209] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.024980] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.055308] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.064272] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.068476] TCP: Hash tables configured (established 65536 bind 65536) [ 1.071942] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.074803] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.077645] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.081341] NET: Registered protocol family 1 [ 1.083703] RPC: Registered named UNIX socket transport module. [ 1.085974] RPC: Registered udp transport module. [ 1.087507] RPC: Registered tcp transport module. [ 1.089207] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.092021] NET: Registered protocol family 44 [ 1.093599] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.095559] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.098033] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.102124] PCI: CLS 0 bytes, default 64 [ 1.105428] Unpacking initramfs... [ 2.737441] debug: unmapping init [mem 0xffff8fd57cc54000-0xffff8fd57ffbffff] [ 2.745325] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.747428] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.749969] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.440120] Initialise system trusted keyrings [ 3.441963] Key type blacklist registered [ 3.453196] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.468062] zbud: loaded [ 3.471349] *** VALIDATE nfs *** [ 3.472449] *** VALIDATE nfs4 *** [ 3.473873] pstore: using deflate compression [ 3.480866] Platform Keyring initialized [ 3.657878] NET: Registered protocol family 38 [ 3.659779] Key type asymmetric registered [ 3.662251] Asymmetric key parser 'x509' registered [ 3.664943] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.669418] io scheduler mq-deadline registered [ 3.672344] io scheduler kyber registered [ 3.674107] io scheduler bfq registered [ 3.676640] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.681352] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.685554] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.688592] ACPI: Power Button [PWRF] [ 3.803634] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.926932] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.140631] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.236736] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.508789] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.548209] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.587555] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.593295] Non-volatile memory driver v1.3 [ 4.599901] Linux agpgart interface v0.103 [ 4.634161] virtio_blk virtio1: [vda] 133912 512-byte logical blocks (68.6 MB/65.4 MiB) [ 4.641553] vda: detected capacity change from 0 to 68562944 [ 4.656158] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.662659] vdb: detected capacity change from 0 to 1073741824 [ 4.697273] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.700976] vdc: detected capacity change from 0 to 2621440000 [ 4.724695] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.732080] vdd: detected capacity change from 0 to 2621440000 [ 4.762069] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.768609] vde: detected capacity change from 0 to 4294967296 [ 4.793213] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.800043] vdf: detected capacity change from 0 to 4294967296 [ 4.810277] libphy: Fixed MDIO Bus: probed [ 4.821601] usbcore: registered new interface driver usbserial_generic [ 4.823905] usbserial: USB Serial support registered for generic [ 4.828374] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.837061] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.838962] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.841615] mousedev: PS/2 mouse device common for all mice [ 4.845618] rtc_cmos 00:05: RTC can wake from S4 [ 4.848114] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.850819] rtc_cmos 00:05: registered as rtc0 [ 4.861071] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.864558] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.867138] intel_pstate: CPU model not supported [ 4.880094] hid: raw HID events driver (C) Jiri Kosina [ 4.885720] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.886844] usbcore: registered new interface driver usbhid [ 4.893600] usbhid: USB HID core driver [ 4.895062] drop_monitor: Initializing network drop monitor service [ 4.905832] Initializing XFRM netlink socket [ 4.907885] NET: Registered protocol family 10 [ 4.912432] Segment Routing with IPv6 [ 4.913944] NET: Registered protocol family 17 [ 4.915967] mpls_gso: MPLS GSO support [ 4.922214] RAS: Correctable Errors collector initialized. [ 4.924429] AVX version of gcm_enc/dec engaged. [ 4.929302] AES CTR mode by8 optimization enabled [ 5.138496] sched_clock: Marking stable (5138466396, 0)->(6240871522, -1102405126) [ 5.144465] registered taskstats version 1 [ 5.148790] Loading compiled-in X.509 certificates [ 5.155389] zswap: loaded using pool lzo/zbud [ 5.217988] Key type big_key registered [ 5.247461] Key type encrypted registered [ 5.248852] ima: No TPM chip found, activating TPM-bypass! [ 5.270663] ima: Allocated hash algorithm: sha1 [ 5.272108] ima: No architecture policies found [ 5.273542] evm: Initialising EVM extended attributes: [ 5.281546] evm: security.selinux [ 5.284916] evm: security.ima [ 5.289142] evm: security.capability [ 5.290528] evm: HMAC attrs: 0x1 [ 5.294172] rtc_cmos 00:05: setting system clock to 2025-10-24 05:19:32 UTC (1761283172) [ 5.303308] debug: unmapping init [mem 0xffffffffb7403000-0xffffffffb75fffff] [ 5.312084] debug: unmapping init [mem 0xffffffffb6182000-0xffffffffb6458fff] [ 5.330099] Write protecting the kernel read-only data: 28672k [ 5.338095] debug: unmapping init [mem 0xffffffffb4803000-0xffffffffb49fffff] [ 5.340494] debug: unmapping init [mem 0xffffffffb5114000-0xffffffffb51fffff] [ 5.438257] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.458986] systemd[1]: Detected virtualization kvm. [ 5.463107] systemd[1]: Detected architecture x86-64. [ 5.470063] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.507727] systemd[1]: No hostname configured. [ 5.510996] systemd[1]: Set hostname to . [ 5.517381] random: systemd: uninitialized urandom read (16 bytes read) [ 5.524202] systemd[1]: Initializing machine ID from random generator. [ 5.656025] random: ln: uninitialized urandom read (6 bytes read) [ 5.879842] random: systemd: uninitialized urandom read (16 bytes read) [ 5.886978] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.894974] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.903430] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.277389] device-mapper: uevent: version 1.0.3 [ 7.279408] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.064645] virtio_net virtio0 ens2: renamed from eth0 [ 9.151862] random: fast init done [ 10.186239] scsi host0: ata_piix [ 10.275089] scsi host1: ata_piix [ 10.276636] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.278781] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.803446] random: crng init done [ 13.805550] random: 7 urandom warning(s) missed due to ratelimiting [ 15.714993] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.896274] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.815997] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.551631] SELinux: Disabled at runtime. [ 19.698251] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.714114] systemd[1]: Detected virtualization kvm. [ 19.717754] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.252415] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.260146] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.268057] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.279973] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.290190] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.304414] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.330120] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ 21.777774] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 22.883636] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.630333] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.832070] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.332847] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.466801] EDAC sbridge: Ver: 1.1.2 [ 28.538467] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 29.139802] NFS: Registering the id_resolver key type [ 29.141154] Key type id_resolver registered [ 29.147185] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Hostname Service... Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg233-server login: [ 56.922210] libcfs: loading out-of-tree module taints kernel. [ 56.936762] Key type ._llcrypt registered [ 56.938262] Key type .llcrypt registered [ 56.986368] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 65.348141] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 65.996572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 66.004393] alg: No test for adler32 (adler32-zlib) [ 67.009417] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 67.342055] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 68.960191] Key type lgssc registered [ 69.681911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 77.105061] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 77.552790] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 80.971127] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.059625] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 87.219649] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.373626] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 98.495646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.526557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 98.540334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 99.662361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 99.679893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 99.730515] Lustre: lustre-MDT0000: new disk, initializing [ 99.782222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 99.795646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 101.488944] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 107.487227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.522133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 107.560573] Lustre: 6489:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 107.582191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 107.584485] Lustre: Skipped 1 previous similar message [ 107.628576] Lustre: lustre-MDT0001: new disk, initializing [ 107.656183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 107.670167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 107.674224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 109.163519] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.494349] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 115.066276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 115.098922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 115.205138] Lustre: lustre-OST0000: new disk, initializing [ 115.207751] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 115.229508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 117.210296] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 117.427242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 117.431752] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 117.458264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 122.690907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 122.723024] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 122.764186] Lustre: lustre-OST0001: new disk, initializing [ 122.767112] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 122.791624] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 124.787818] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 125.100457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 125.104414] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 125.129552] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 131.183844] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 136.009257] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 143.074473] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing check_logdir /tmp/testlogs/ [ 144.650790] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing yml_node [ 146.349269] Lustre: DEBUG MARKER: Client: 2.16.59.47 [ 147.234201] Lustre: DEBUG MARKER: MDS: 2.16.59.47 [ 148.177871] Lustre: DEBUG MARKER: OSS: 2.16.59.47 [ 148.806092] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 01:21:55 EDT 2025 [ 155.177654] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 155.732109] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 169.441382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 169.446554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 169.455673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 171.490083] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 171.491312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 171.502759] Lustre: Skipped 3 previous similar messages [ 176.611706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 176.614285] Lustre: Skipped 3 previous similar messages [ 181.728747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 181.731719] Lustre: Skipped 3 previous similar messages [ 182.240196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 182.283427] LustreError: 12193:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 182.316087] Lustre: server umount lustre-MDT0000 complete [ 184.036880] LustreError: 6480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761283351 with bad export cookie 9705621417215597801 [ 184.038446] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 184.042118] LustreError: 6480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 186.848843] LustreError: 8094:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 186.849071] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 186.849332] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 186.854817] LustreError: 8094:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 186.865336] Lustre: Skipped 3 previous similar messages [ 189.857133] LustreError: 8094:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 189.863739] LustreError: 8094:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 190.216789] LustreError: 12398:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 190.219533] LustreError: 12398:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 190.328739] Lustre: server umount lustre-MDT0001 complete [ 192.100380] LustreError: 12601:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 192.103547] LustreError: 12601:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 192.134249] Lustre: server umount lustre-OST0000 complete [ 193.907207] Lustre: server umount lustre-OST0001 complete [ 196.476018] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 199.985103] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 204.410905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 207.227701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 209.979342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 213.165306] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 216.318972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 216.355991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.485269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 216.507323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 216.561883] Lustre: lustre-MDT0000: new disk, initializing [ 216.617381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 216.635359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 218.435592] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 223.400252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 223.424130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 223.467713] Lustre: 14982:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 223.600714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 223.637883] Lustre: 15782:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 223.642525] Lustre: 15782:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 223.648452] Lustre: 15782:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 223.652042] Lustre: 15782:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 223.655108] Lustre: 15782:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 223.658741] Lustre: 15782:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 225.083306] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 228.351054] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 229.598249] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 231.956980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 231.982743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 232.085284] Lustre: lustre-OST0000: new disk, initializing [ 232.087269] Lustre: Skipped 1 previous similar message [ 232.088988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 232.090758] Lustre: Skipped 2 previous similar messages [ 232.119740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 232.121682] Lustre: Skipped 1 previous similar message [ 233.727260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 233.731966] Lustre: Skipped 1 previous similar message [ 233.734682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 233.794987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 234.318935] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 237.723298] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 239.305922] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 239.406369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 241.015513] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 241.113395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 244.193559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 244.196365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 244.197374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 244.202165] Lustre: Skipped 1 previous similar message [ 244.204807] Lustre: Skipped 3 previous similar messages [ 247.920141] LustreError: 17922:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 247.923159] LustreError: 17922:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 247.996178] Lustre: server umount lustre-OST0000 complete [ 254.434969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.435050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 254.439715] Lustre: Skipped 1 previous similar message [ 254.443124] LustreError: Skipped 1 previous similar message [ 255.620058] LustreError: 18123:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 255.623310] LustreError: 18123:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 255.671829] Lustre: server umount lustre-MDT0000 complete [ 257.161654] LustreError: 14972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761283424 with bad export cookie 9705621417215599348 [ 257.165045] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 257.166171] LustreError: 14972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 257.342490] Lustre: server umount lustre-MDT0001 complete [ 259.028261] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:23:45 (1761283425) [ 260.057328] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 260.789143] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:23:46 (1761283426) [ 263.750764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 263.938240] LustreError: 18995:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 263.976568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 265.693720] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 269.099843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 270.691517] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 272.794798] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 274.207980] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 276.947624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 279.497193] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 281.819277] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 300.004883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 300.005438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.006508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 300.006514] Lustre: Skipped 3 previous similar messages [ 300.017654] Lustre: Skipped 1 previous similar message [ 301.294144] LustreError: 21262:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 301.296288] LustreError: 21262:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 301.364081] Lustre: server umount lustre-OST0000 complete [ 308.897408] Lustre: server umount lustre-MDT0000 complete [ 310.373337] LustreError: 19020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761283477 with bad export cookie 9705621417215600272 [ 310.376650] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 310.377919] LustreError: 19020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 310.542079] Lustre: server umount lustre-MDT0001 complete [ 314.917433] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 316.183295] Key type lgssc unregistered [ 316.341577] LNet: 22197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 316.345625] LNetError: 22197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 316.356799] LNet: Removed LNI 192.168.202.133@tcp [ 316.779197] Key type .llcrypt unregistered [ 316.780937] Key type ._llcrypt unregistered [ 327.303219] Key type ._llcrypt registered [ 327.305304] Key type .llcrypt registered [ 327.379923] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 327.761367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 327.809476] alg: No test for adler32 (adler32-zlib) [ 328.704122] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 328.805857] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 330.401057] Key type lgssc registered [ 330.918969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 334.584562] loop: module loaded [ 334.625825] loop0: detected capacity change from 0 to 204800000 [ 334.702469] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 334.708761] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 334.860972] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 334.886882] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 337.335581] loop0: detected capacity change from 0 to 204800000 [ 337.418038] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 337.423524] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 337.429536] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 337.483941] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 339.354295] loop0: detected capacity change from 0 to 204800000 [ 341.671177] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 341.699278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 341.709221] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 342.823155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 342.848576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 342.891982] Lustre: lustre-MDT0000: new disk, initializing [ 342.951685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 342.964084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 342.985926] Lustre: 24309:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 466, rollback = 2 [ 342.990558] Lustre: 24309:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 342.994096] Lustre: 24309:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 2/466/0 [ 342.997408] Lustre: 24309:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 343.000808] Lustre: 24309:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 343.003665] Lustre: 24309:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 344.605496] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 349.006956] loop1: detected capacity change from 0 to 204800000 [ 351.365592] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 351.396637] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 351.530032] Lustre: lustre-OST07c6: new disk, initializing [ 351.533217] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 351.535654] Lustre: Skipped 1 previous similar message [ 351.567792] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 353.928845] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 355.434773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 355.440044] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 355.457301] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 357.461249] Lustre: 25345:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 357.465649] Lustre: 25345:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 2 previous similar messages [ 357.469424] Lustre: 25345:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 357.473073] Lustre: 25345:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 357.476800] Lustre: 25345:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 357.480109] Lustre: 25345:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 357.484485] Lustre: 25345:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 357.488141] Lustre: 25345:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 357.492475] Lustre: 25345:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 357.496228] Lustre: 25345:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 357.499382] Lustre: 25345:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 357.503617] Lustre: 25345:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 374.170461] Lustre: 24335:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 497, rollback = 2 [ 374.174485] Lustre: 24335:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 374.176765] Lustre: 24335:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 374.179329] Lustre: 24335:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 374.181549] Lustre: 24335:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 374.184711] Lustre: 24335:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 377.868616] Lustre: 24336:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 497, rollback = 2 [ 377.872704] Lustre: 24336:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 377.878958] Lustre: 24336:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 377.882618] Lustre: 24336:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 377.888352] Lustre: 24336:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 377.894801] Lustre: 24336:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 377.900844] Lustre: 24336:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 377.905183] Lustre: 24336:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 377.909731] Lustre: 24336:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 377.913681] Lustre: 24336:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 377.918278] Lustre: 24336:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 377.922629] Lustre: 24336:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 384.763964] Lustre: 24334:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 497, rollback = 2 [ 384.768405] Lustre: 24334:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 3 previous similar messages [ 384.771768] Lustre: 24334:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 384.775223] Lustre: 24334:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 384.779100] Lustre: 24334:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 384.782674] Lustre: 24334:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 384.786354] Lustre: 24334:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 384.790274] Lustre: 24334:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 384.794092] Lustre: 24334:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 384.799515] Lustre: 24334:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 384.807675] Lustre: 24334:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 384.812564] Lustre: 24334:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 394.601591] Lustre: 24335:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 497, rollback = 2 [ 394.606169] Lustre: 24335:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 5 previous similar messages [ 394.609587] Lustre: 24335:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 394.612961] Lustre: 24335:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 394.616394] Lustre: 24335:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 394.619767] Lustre: 24335:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 394.623721] Lustre: 24335:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 394.627564] Lustre: 24335:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 394.631159] Lustre: 24335:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 394.634480] Lustre: 24335:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 394.638168] Lustre: 24335:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 394.641553] Lustre: 24335:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 441.071223] LustreError: 26525:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 441.139455] Lustre: server umount lustre-OST07c6 complete [ 454.117636] LustreError: 27025:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 454.120469] LustreError: 27025:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 454.225655] Lustre: server umount lustre-MDT0000 complete [ 460.885644] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 461.996868] Key type lgssc unregistered [ 462.123352] LNet: 27858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 462.125607] LNetError: 27858:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 462.132345] LNet: Removed LNI 192.168.202.133@tcp [ 462.463995] Key type .llcrypt unregistered [ 462.465747] Key type ._llcrypt unregistered [ 470.963259] Key type ._llcrypt registered [ 470.964591] Key type .llcrypt registered [ 471.011195] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 471.322606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 471.330278] alg: No test for adler32 (adler32-zlib) [ 472.204301] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 472.305222] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 473.896218] Key type lgssc registered [ 474.303638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 477.265760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 477.273061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 478.446502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.848376] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 482.919671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 483.046725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 484.395408] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 486.393566] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 487.703129] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 490.056732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 490.159754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.997533] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 493.835346] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 503.463101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 506.199162] LustreError: 31306:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 508.899954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 508.900122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.903513] LustreError: Skipped 1 previous similar message [ 508.910595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 508.913841] Lustre: Skipped 1 previous similar message [ 514.018062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 514.020568] Lustre: Skipped 1 previous similar message [ 514.863084] LustreError: 31557:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 514.922880] Lustre: server umount lustre-OST0000 complete [ 519.137967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.138628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 519.140932] Lustre: Skipped 2 previous similar messages [ 519.141140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 519.142597] Lustre: Skipped 1 previous similar message [ 522.368342] LustreError: 31758:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 522.370338] LustreError: 31758:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 522.399888] Lustre: server umount lustre-MDT0000 complete [ 523.684626] LustreError: 29067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761283690 with bad export cookie 17581575035126631030 [ 523.688263] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 523.689593] LustreError: 29067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 523.751965] LustreError: 31960:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 523.753954] LustreError: 31960:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 523.832151] Lustre: server umount lustre-MDT0001 complete [ 527.416586] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 528.520448] Key type lgssc unregistered [ 528.655409] LNet: 32491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 528.658514] LNetError: 32491:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 528.671301] LNet: Removed LNI 192.168.202.133@tcp [ 528.963991] Key type .llcrypt unregistered [ 528.965324] Key type ._llcrypt unregistered [ 535.967093] Key type ._llcrypt registered [ 535.968518] Key type .llcrypt registered [ 536.001341] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 536.385017] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 536.391346] alg: No test for adler32 (adler32-zlib) [ 537.242153] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 537.327887] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 538.904223] Key type lgssc registered [ 539.271451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 541.931805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 541.938600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 543.088334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 544.330373] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 547.215262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 547.335410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 548.708125] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 550.527213] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 551.751945] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 553.952486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 554.060661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.838116] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 557.621579] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 568.427372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 578.017221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 578.019352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.023071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 579.042644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 579.043443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.043817] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 583.151214] LustreError: 35987:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 583.210650] Lustre: server umount lustre-OST0000 complete [ 588.256980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 588.259467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.263661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 588.265465] Lustre: Skipped 1 previous similar message [ 590.592220] LustreError: 36190:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 590.596169] LustreError: 36190:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 590.627195] Lustre: server umount lustre-MDT0000 complete [ 591.962557] LustreError: 33698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761283759 with bad export cookie 14092634629603772076 [ 591.965193] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.967352] LustreError: 33698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 592.039094] LustreError: 36392:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 592.041776] LustreError: 36392:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 592.135750] Lustre: server umount lustre-MDT0001 complete [ 595.841539] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 596.883341] Key type lgssc unregistered [ 597.029324] LNet: 36924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.031791] LNetError: 36924:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 597.041290] LNet: Removed LNI 192.168.202.133@tcp [ 597.307908] Key type .llcrypt unregistered [ 597.308854] Key type ._llcrypt unregistered [ 605.906664] Key type ._llcrypt registered [ 605.907926] Key type .llcrypt registered [ 605.950043] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 606.430394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 606.438420] alg: No test for adler32 (adler32-zlib) [ 607.319641] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 607.412694] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 609.000132] Key type lgssc registered [ 609.355329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 611.929147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 611.935254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.078894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 614.364273] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 617.294142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 617.411715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 618.658593] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 620.380244] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 621.586898] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 623.864785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 623.976519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 625.930533] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 627.708068] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 638.312114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 643.040811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 643.042638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 643.045829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 643.550403] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 643.555863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 643.559739] Lustre: Skipped 1 previous similar message [ 646.781357] LustreError: 40568:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 646.811494] Lustre: server umount lustre-MDT0000 complete [ 648.126198] LustreError: 38277:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761283815 with bad export cookie 16424995839009146305 [ 648.131362] LustreError: 38277:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 648.132421] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 648.662471] LustreError: 40457:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 648.668957] LustreError: 40457:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 648.670620] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 648.673688] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 648.675812] Lustre: Skipped 2 previous similar messages [ 653.782120] LustreError: 38294:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.782243] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 653.788767] LustreError: 38294:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 654.269408] LustreError: 40771:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 654.271732] LustreError: 40771:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 654.352333] Lustre: server umount lustre-MDT0001 complete [ 672.301219] LustreError: 40996:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 672.302887] LustreError: 40996:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 672.365676] Lustre: server umount lustre-OST0000 complete [ 677.783889] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 678.856523] Key type lgssc unregistered [ 678.980500] LNet: 41834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.983519] LNetError: 41834:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 678.991424] LNet: Removed LNI 192.168.202.133@tcp [ 679.289052] Key type .llcrypt unregistered [ 679.290053] Key type ._llcrypt unregistered [ 688.042489] Key type ._llcrypt registered [ 688.043443] Key type .llcrypt registered [ 688.080191] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 688.387109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 688.395676] alg: No test for adler32 (adler32-zlib) [ 689.258021] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 689.343957] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 690.920185] Key type lgssc registered [ 691.276494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 693.967728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 693.974294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 695.125561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 696.386874] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 699.307498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 699.424716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 700.723172] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 702.438409] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 703.585840] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 705.822126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 705.920784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 707.680238] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 709.409157] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 720.298120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 729.568907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 729.571181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 729.575561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 730.078083] Lustre: lustre-OST0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 730.593655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 730.595364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 733.166084] LustreError: 45484:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 733.228721] Lustre: server umount lustre-OST0000 complete [ 746.035584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 746.036550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 746.039557] Lustre: Skipped 1 previous similar message [ 746.041806] Lustre: Skipped 2 previous similar messages [ 751.073289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 751.076249] Lustre: Skipped 2 previous similar messages [ 752.187607] LustreError: 45860:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 752.189766] LustreError: 45860:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 752.220132] Lustre: server umount lustre-MDT0000 complete [ 753.478496] LustreError: 43196:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761283920 with bad export cookie 11262549007796306278 [ 753.482764] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 753.483728] LustreError: 43196:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 759.612141] LustreError: 46061:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 759.615322] LustreError: 46061:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 759.691651] Lustre: server umount lustre-MDT0001 complete [ 763.226517] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 764.263920] Key type lgssc unregistered [ 764.399385] LNet: 46593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.402255] LNetError: 46593:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 764.410304] LNet: Removed LNI 192.168.202.133@tcp [ 764.716676] Key type .llcrypt unregistered [ 764.718308] Key type ._llcrypt unregistered [ 773.510567] Key type ._llcrypt registered [ 773.511621] Key type .llcrypt registered [ 773.548401] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 773.874125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 773.881512] alg: No test for adler32 (adler32-zlib) [ 774.734933] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 774.818957] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 776.400204] Key type lgssc registered [ 776.748799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 779.396432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 779.403963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 780.561462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 781.808986] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 784.570901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 784.686658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 785.921178] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 787.614749] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 788.785645] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 790.909439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 791.034751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 792.827787] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 793.267804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 794.489668] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 797.476249] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 798.641427] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 809.368715] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-24 1h32m55s [ 809.884417] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 810.973995] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 811.989715] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 815.078248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 815.079597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 815.082996] Lustre: Skipped 2 previous similar messages [ 815.087310] Lustre: Skipped 2 previous similar messages [ 816.772967] LustreError: 50531:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 816.805505] Lustre: server umount lustre-MDT0000 complete [ 817.987947] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 818.133063] LustreError: 47962:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 818.138769] LustreError: 47962:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 820.324897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 820.373464] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 820.477270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 820.504704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 821.543426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 821.897059] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 823.195430] Lustre: DEBUG MARKER: Wait for df (35963) ... [ 823.723715] Lustre: DEBUG MARKER: done [ 825.312918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 825.315042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 825.318615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 825.321575] Lustre: Skipped 3 previous similar messages [ 825.825685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 826.672089] LustreError: 51579:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 826.674766] LustreError: 51579:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 826.752451] Lustre: server umount lustre-OST0000 complete [ 830.076370] LustreError: 51780:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 830.079432] LustreError: 51780:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 830.103976] Lustre: server umount lustre-MDT0000 complete [ 831.262355] LustreError: 47949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761283998 with bad export cookie 3603243906246171786 [ 831.265445] LustreError: 47949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 831.265936] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 837.401622] LustreError: 51984:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 837.403697] LustreError: 51984:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 837.487363] Lustre: server umount lustre-MDT0001 complete [ 840.786032] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 841.731271] Key type lgssc unregistered [ 841.867486] LNet: 52515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.870548] LNetError: 52515:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.879403] LNet: Removed LNI 192.168.202.133@tcp [ 842.171066] Key type .llcrypt unregistered [ 842.172123] Key type ._llcrypt unregistered [ 853.713100] Key type ._llcrypt registered [ 853.714064] Key type .llcrypt registered [ 853.747269] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 859.836724] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 860.168045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 860.208786] alg: No test for adler32 (adler32-zlib) [ 861.072454] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 861.157576] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 862.744166] Key type lgssc registered [ 863.114953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 866.238988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 868.512387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 870.673311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 872.935263] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 875.298944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 875.318789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 875.323927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 876.402833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 876.414327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 876.449870] Lustre: lustre-MDT0000: new disk, initializing [ 876.475179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 876.483088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 877.795306] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 881.815055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 881.836254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 881.869276] Lustre: 56437:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 881.895523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 881.897771] Lustre: Skipped 1 previous similar message [ 881.939647] Lustre: lustre-MDT0001: new disk, initializing [ 881.974930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 881.987238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 881.991080] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 882.023604] Lustre: 57237:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 882.026107] Lustre: 57237:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 882.028421] Lustre: 57237:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 882.030188] Lustre: 57237:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 882.031816] Lustre: 57237:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 882.033629] Lustre: 57237:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 883.298702] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 886.372483] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 887.521419] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 889.633166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 889.654852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 889.734769] Lustre: lustre-OST0000: new disk, initializing [ 889.736455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 889.761495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 890.554964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 890.558526] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 890.600210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 891.600991] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 894.654781] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 896.059976] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 896.146699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 897.479127] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 897.564081] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 901.089853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 901.093000] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 901.097397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 901.099221] Lustre: Skipped 1 previous similar message [ 904.304309] LustreError: 59377:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 904.381326] Lustre: server umount lustre-OST0000 complete [ 906.209701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 906.210205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 906.213812] Lustre: Skipped 2 previous similar messages [ 906.216858] Lustre: Skipped 1 previous similar message [ 911.329842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 911.331582] Lustre: Skipped 1 previous similar message [ 916.449245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 916.452352] Lustre: Skipped 1 previous similar message [ 920.032158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 920.066990] LustreError: 59580:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 920.069212] LustreError: 59580:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 920.098630] Lustre: server umount lustre-MDT0000 complete [ 921.401419] LustreError: 56427:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284088 with bad export cookie 13642740227510017049 [ 921.405233] LustreError: 56427:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 921.406123] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 921.477310] LustreError: 59782:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 921.479137] LustreError: 59782:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 921.564475] Lustre: server umount lustre-MDT0001 complete [ 924.547705] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 01:34:50 (1761284090) [ 926.927996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.104594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 928.514237] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 931.470777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 931.601056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 932.923414] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 934.810134] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 936.057733] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 938.424666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 940.557489] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 942.492672] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 955.782734] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-24 1h35m22s [ 956.345078] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 958.755371] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 959.623586] Lustre: DEBUG MARKER: Creating a test file and stat it [ 959.647525] LustreError: 62971:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 979.937431] Lustre: lustre-MDT0000: Client 73d67543-900f-4db0-a206-77fbe1ab83c1 (at 192.168.202.33@tcp) reconnecting [ 999.688092] LustreError: 62971:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1000.494425] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1001.378959] Lustre: DEBUG MARKER: done [ 1004.001798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1004.001995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.003516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1004.004701] LustreError: Skipped 1 previous similar message [ 1004.009041] Lustre: Skipped 1 previous similar message [ 1009.198199] LustreError: 63479:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1009.200298] LustreError: 63479:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1009.260995] Lustre: server umount lustre-OST0000 complete [ 1013.728924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1013.732894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.737250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1013.739193] Lustre: Skipped 3 previous similar messages [ 1016.638742] LustreError: 63681:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1016.641321] LustreError: 63681:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1016.669718] Lustre: server umount lustre-MDT0000 complete [ 1017.956188] LustreError: 60432:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284185 with bad export cookie 13642740227510017959 [ 1017.959535] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1017.959970] LustreError: 60432:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1018.105627] Lustre: server umount lustre-MDT0001 complete [ 1022.018724] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1023.157823] Key type lgssc unregistered [ 1023.286375] LNet: 64415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.289069] LNetError: 64415:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.298311] LNet: Removed LNI 192.168.202.133@tcp [ 1023.616378] Key type .llcrypt unregistered [ 1023.617588] Key type ._llcrypt unregistered [ 1034.752065] Key type ._llcrypt registered [ 1034.752992] Key type .llcrypt registered [ 1034.792654] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 1041.057156] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1041.374076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1041.404203] alg: No test for adler32 (adler32-zlib) [ 1042.253502] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 1042.328118] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1043.904147] Key type lgssc registered [ 1044.247198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1047.183073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1049.381069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1051.321846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1053.335067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1055.530025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1055.549170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1055.553973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1056.634391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1056.645231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1056.677608] Lustre: lustre-MDT0000: new disk, initializing [ 1056.699423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1056.705345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1057.937786] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1061.998262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1062.020949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1062.049283] Lustre: 68334:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1062.068667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1062.070653] Lustre: Skipped 1 previous similar message [ 1062.104395] Lustre: lustre-MDT0001: new disk, initializing [ 1062.133653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1062.144441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1062.147508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1062.173377] Lustre: 69134:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 1062.177655] Lustre: 69134:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 1062.179933] Lustre: 69134:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 1062.182348] Lustre: 69134:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 1062.186628] Lustre: 69134:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 1062.189289] Lustre: 69134:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 1063.375694] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1066.362489] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1067.554317] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1069.745388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1069.767089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1069.862483] Lustre: lustre-OST0000: new disk, initializing [ 1069.864662] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1069.889570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1071.592697] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1074.687308] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1076.063950] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1076.729281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1076.732561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1076.762681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1077.161222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1078.439320] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1078.520571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1081.826668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1081.828432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.830590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1081.835777] Lustre: Skipped 1 previous similar message [ 1085.231121] LustreError: 71284:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1085.308409] Lustre: server umount lustre-OST0000 complete [ 1086.946314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1086.946932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1086.953142] Lustre: Skipped 1 previous similar message [ 1086.954695] Lustre: Skipped 1 previous similar message [ 1092.065355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1092.068345] Lustre: Skipped 2 previous similar messages [ 1092.735781] LustreError: 71485:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1092.737449] LustreError: 71485:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1092.765774] Lustre: server umount lustre-MDT0000 complete [ 1094.083670] LustreError: 68327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284261 with bad export cookie 10183412444397447912 [ 1094.088250] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1094.089616] LustreError: 68327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1094.164548] LustreError: 71687:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1094.166466] LustreError: 71687:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1094.255129] Lustre: server umount lustre-MDT0001 complete [ 1097.116421] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 01:37:43 (1761284263) [ 1097.646273] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1098.247563] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 01:37:44 (1761284264) [ 1102.391202] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1104.078577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1104.269194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1104.617450] Lustre: Failing over lustre-MDT0000 [ 1104.619501] LustreError: 72761:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1104.622870] LustreError: 72761:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1104.625579] LustreError: 72761:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1104.687751] LustreError: 72813:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1104.689799] LustreError: 72813:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1104.763616] Lustre: server umount lustre-MDT0000 complete [ 1106.921279] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:37:53 (1761284273) [ 1109.231426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1109.394515] LustreError: 73382:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1109.400511] LustreError: 73382:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1109.410639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1110.827662] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1113.808343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1115.283577] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1117.189825] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1118.391759] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1120.679279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1120.790390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1120.792807] Lustre: Skipped 1 previous similar message [ 1122.693910] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1124.499285] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1137.571377] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1142.909949] LustreError: 75749:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1142.912646] LustreError: 75749:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1142.943233] Lustre: server umount lustre-MDT0000 complete [ 1144.289798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.291568] LustreError: 75542:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1144.295807] Lustre: Skipped 2 previous similar messages [ 1144.302051] LustreError: 75542:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 1144.409680] LustreError: 75002:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284311 with bad export cookie 10183412444397449081 [ 1144.413127] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1144.414472] LustreError: 75002:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1146.680423] Lustre: server umount lustre-MDT0001 complete [ 1148.240277] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1150.727119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1154.257226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1154.528796] LustreError: 66549:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fb21aede x1846840350800000/t0(0) o250->MGC192.168.202.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1154.635335] LustreError: 74990:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1154.664481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.697691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1155.962084] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1158.880760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1158.976210] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1160.406595] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1162.200621] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1163.374802] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1164.134107] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1165.136122] Lustre: 66551:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761284316/real 1761284316] req@0000000047867088 x1846840350799744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761284332 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1170.385114] Lustre: 66552:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761284321/real 1761284321] req@00000000c5c85030 x1846840350799872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761284337 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1174.498769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1174.499151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1174.501351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1174.504074] Lustre: Skipped 1 previous similar message [ 1174.506798] Lustre: Skipped 2 previous similar messages [ 1176.635890] LustreError: 78208:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1176.637954] LustreError: 78208:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1176.663626] Lustre: server umount lustre-MDT0000 complete [ 1177.926444] LustreError: 75002:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284345 with bad export cookie 10183412444397450971 [ 1177.931092] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1178.079429] Lustre: server umount lustre-MDT0001 complete [ 1180.262990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1183.403676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1188.768890] LustreError: 66549:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fc84492c x1846840350821376/t0(0) o250->MGC192.168.202.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1188.879502] LustreError: 77522:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1188.903808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1188.906438] Lustre: Skipped 1 previous similar message [ 1188.922410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1190.271269] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1193.196273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1193.287847] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.294397] Lustre: Skipped 1 previous similar message [ 1194.342270] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1194.344898] Lustre: Skipped 1 previous similar message [ 1194.636101] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1195.361091] Lustre: 66552:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761284346/real 1761284346] req@0000000007360be6 x1846840350820992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761284362 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1196.519275] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1197.776297] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1198.432079] Lustre: 66553:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761284349/real 1761284349] req@00000000f0ba80ac x1846840350821120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761284365 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1199.587773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1199.589753] Lustre: Skipped 2 previous similar messages [ 1204.704248] Lustre: 66551:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761284355/real 1761284355] req@00000000c57e7db7 x1846840350821504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761284371 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1204.712480] Lustre: 66551:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1205.689828] LustreError: 80661:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1205.691874] LustreError: 80661:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1205.719345] Lustre: server umount lustre-MDT0000 complete [ 1206.974493] LustreError: 75002:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284374 with bad export cookie 10183412444397452343 [ 1206.978546] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1206.980103] LustreError: 75002:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1207.112726] Lustre: server umount lustre-MDT0001 complete [ 1209.312844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1210.477585] Lustre: DEBUG MARKER: files compared the same [ 1221.491698] Lustre: server umount lustre-OST0000 complete [ 1226.908169] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1227.955147] Key type lgssc unregistered [ 1228.075344] LNet: 82292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1228.078810] LNetError: 82292:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1228.088301] LNet: Removed LNI 192.168.202.133@tcp [ 1228.370459] Key type .llcrypt unregistered [ 1228.371450] Key type ._llcrypt unregistered [ 1238.196568] Key type ._llcrypt registered [ 1238.198059] Key type .llcrypt registered [ 1238.241503] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1238.612725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1238.657846] alg: No test for adler32 (adler32-zlib) [ 1239.531885] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 1239.628378] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1241.216197] Key type lgssc registered [ 1241.608468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1244.357747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1244.364509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1245.524429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1246.865234] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug +malloc all [ 1249.938652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1250.054659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1251.459990] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug +malloc all [ 1253.391805] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1254.698344] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1257.019882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1257.109153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1259.039629] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug +malloc all [ 1260.939215] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1270.436583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1275.872911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1275.872966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.875775] LustreError: Skipped 1 previous similar message [ 1275.880364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1275.883023] Lustre: Skipped 1 previous similar message [ 1275.948085] LustreError: 85934:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1276.032939] Lustre: server umount lustre-OST0000 complete [ 1279.421036] LustreError: 86137:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1279.423123] LustreError: 86137:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1279.453949] Lustre: server umount lustre-MDT0000 complete [ 1280.757859] LustreError: 85285:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284447 with bad export cookie 14280927237169951476 [ 1280.759908] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1280.761965] LustreError: 85285:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1280.820696] LustreError: 86339:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1280.822688] LustreError: 86339:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1280.891449] Lustre: server umount lustre-MDT0001 complete [ 1284.449272] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1285.416248] Key type lgssc unregistered [ 1285.538395] LNet: 86870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1285.542141] LNetError: 86870:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1285.555347] LNet: Removed LNI 192.168.202.133@tcp [ 1285.845473] Key type .llcrypt unregistered [ 1285.846753] Key type ._llcrypt unregistered [ 1294.791025] Key type ._llcrypt registered [ 1294.791960] Key type .llcrypt registered [ 1294.830609] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1295.218305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1295.249617] alg: No test for adler32 (adler32-zlib) [ 1296.123291] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 1296.212416] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1297.800262] Key type lgssc registered [ 1298.173208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1300.842506] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1300.850538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1316.320478] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1321.440171] LustreError: 88217:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 1348.064231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.508847] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1354.693252] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1361.344464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.416552] Lustre: *** cfs_fail_loc=706, val=0*** [ 1361.757980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.824236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1365.309313] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1368.034472] LustreError: 89191:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1368.062926] LustreError: 89191:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 1372.308527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1372.525642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1375.431811] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1380.176728] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1383.931803] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1387.493779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1387.505895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1387.533224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1388.005751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1388.021200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1388.043412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1388.046616] Lustre: Skipped 1 previous similar message [ 1392.210776] LustreError: 90545:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1392.603871] Lustre: server umount lustre-OST0000 complete [ 1398.242502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1398.249098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1398.252063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1398.273882] Lustre: Skipped 1 previous similar message [ 1402.865191] LustreError: 90747:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1402.867706] LustreError: 90747:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1403.079554] Lustre: server umount lustre-MDT0000 complete [ 1403.367734] LustreError: 89196:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1403.405486] LustreError: 89196:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 1407.630782] LustreError: 88226:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284574 with bad export cookie 13174720069975062638 [ 1407.647279] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1407.888112] LustreError: 90949:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1407.890693] LustreError: 90949:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1408.342960] Lustre: server umount lustre-MDT0001 complete [ 1419.786101] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1422.486800] Key type lgssc unregistered [ 1422.770678] LNet: 91480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.780951] LNetError: 91480:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1422.791976] LNet: Removed LNI 192.168.202.133@tcp [ 1423.427063] Key type .llcrypt unregistered [ 1423.429231] Key type ._llcrypt unregistered [ 1439.187399] Key type ._llcrypt registered [ 1439.188773] Key type .llcrypt registered [ 1439.247117] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1439.892615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1439.905849] alg: No test for adler32 (adler32-zlib) [ 1440.840297] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 1440.970434] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1442.600346] Key type lgssc registered [ 1443.246455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1447.992484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1448.002343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1451.033284] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1455.744508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1456.064318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1458.229781] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1461.217377] LustreError: 93526:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.223265] LustreError: 93526:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 1462.214696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1462.394375] LustreError: 93526:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1462.400827] LustreError: 93526:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 1462.408834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1465.638653] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1466.532509] LustreError: 93526:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1466.549643] LustreError: 93526:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1470.991666] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1471.002304] Lustre: MGS: Client 865a40ef-ca72-4432-8b91-5bb110805a42 (at 0@lo) reconnecting [ 1471.142628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1471.181175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1473.334917] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1486.820164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1486.821632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1486.823117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1486.826242] Lustre: Skipped 1 previous similar message [ 1490.240131] LustreError: 95489:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1490.362774] Lustre: server umount lustre-OST0000 complete [ 1497.060147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.060971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1497.077615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1497.095026] Lustre: Skipped 2 previous similar messages [ 1499.355170] LustreError: 95690:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1499.361149] LustreError: 95690:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1499.378657] Lustre: server umount lustre-MDT0000 complete [ 1501.726062] LustreError: 95891:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1501.728780] LustreError: 95891:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1501.851599] Lustre: server umount lustre-MDT0001 complete [ 1503.993854] LustreError: 96093:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1503.997095] LustreError: 96093:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1504.041773] Lustre: server umount MGS complete [ 1511.038557] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:44:36 (1761284676) [ 1527.720036] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 1533.272439] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1538.114388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1542.116287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1547.359853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1552.293483] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1557.082487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1557.128746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1559.556624] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1566.889632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1566.939443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1567.097780] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1567.148257] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1567.204713] Lustre: lustre-MDT0001: new disk, initializing [ 1567.269161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1567.305319] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1567.308276] Lustre: 100605:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1567.322512] LustreError: 100605:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1567.378365] LustreError: 100607:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 1567.396389] LustreError: 100607:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 1567.879037] LustreError: 100607:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 1567.884028] LustreError: 100607:0:(fld_handler.c:251:fld_server_lookup()) Skipped 4220 previous similar messages [ 1567.897087] LustreError: 100607:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 1567.901345] LustreError: 100607:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 4312 previous similar messages [ 1568.879058] LustreError: 100607:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 1568.892207] LustreError: 100607:0:(fld_handler.c:251:fld_server_lookup()) Skipped 7408 previous similar messages [ 1568.901664] LustreError: 100607:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 1568.910767] LustreError: 100607:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 7316 previous similar messages [ 1570.879097] LustreError: 100607:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 1570.884589] LustreError: 100607:0:(fld_handler.c:251:fld_server_lookup()) Skipped 12094 previous similar messages [ 1570.902980] LustreError: 100607:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 1570.910651] LustreError: 100607:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 12239 previous similar messages [ 1571.439679] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1574.312136] hrtimer: interrupt took 2894792 ns [ 1574.879938] LustreError: 100607:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 1574.927512] LustreError: 100607:0:(fld_handler.c:251:fld_server_lookup()) Skipped 22617 previous similar messages [ 1574.956801] LustreError: 100607:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 1574.983425] LustreError: 100607:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 22472 previous similar messages [ 1582.885246] LustreError: 100607:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 1582.915142] LustreError: 100607:0:(fld_handler.c:251:fld_server_lookup()) Skipped 21400 previous similar messages [ 1582.963520] LustreError: 100607:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 1582.988929] LustreError: 100607:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 21408 previous similar messages [ 1587.873508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1588.027742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1588.693171] Lustre: lustre-OST0000: new disk, initializing [ 1588.702654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1588.828389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.887759] LustreError: 100607:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 1598.924910] LustreError: 100607:0:(fld_handler.c:251:fld_server_lookup()) Skipped 33073 previous similar messages [ 1599.110614] LustreError: 100607:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 1599.126227] LustreError: 100607:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 33442 previous similar messages [ 1599.595265] Lustre: 101617:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000001ee91100 x1846840769156608/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1761284782 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1601.115196] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1614.333853] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1620.463593] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1623.129241] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1623.152677] Lustre: MGS: Client 8360385e-c0fd-4032-b9e7-1ebee42d0ece (at 0@lo) reconnecting [ 1623.236743] Lustre: 99889:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1623.267783] LustreError: 99889:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1623.347955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1623.474286] Lustre: lustre-MDT0000: new disk, initializing [ 1623.668107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1623.707418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1624.738841] Lustre: 102321:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@0000000037d28d82 x1846840769163904/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1761284807 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1628.434645] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1629.796587] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1630.890426] LustreError: 100607:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 1630.915199] LustreError: 100607:0:(fld_handler.c:251:fld_server_lookup()) Skipped 124032 previous similar messages [ 1631.117797] LustreError: 100607:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 1631.144786] LustreError: 100607:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 124685 previous similar messages [ 1632.531634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1632.558955] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1634.021245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1634.025995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1634.075488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1659.873087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1659.876763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1659.901100] Lustre: Skipped 1 previous similar message [ 1659.920510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1663.477107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1663.513664] Lustre: Skipped 1 previous similar message [ 1665.281167] LustreError: 102845:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1665.301330] LustreError: 102845:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1665.550967] Lustre: server umount lustre-OST0000 complete [ 1673.697137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1673.702553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.712947] LustreError: Skipped 1 previous similar message [ 1673.717882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1673.717890] Lustre: Skipped 1 previous similar message [ 1673.801376] Lustre: Skipped 2 previous similar messages [ 1676.430353] LustreError: 103049:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1676.474078] LustreError: 103049:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1676.542573] Lustre: server umount lustre-MDT0000 complete [ 1678.817940] LustreError: 100575:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1678.833032] LustreError: 100575:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 1686.678258] Lustre: server umount lustre-MDT0001 complete [ 1690.289963] Lustre: server umount MGS complete [ 1701.693395] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:47:46 (1761284866) [ 1719.856926] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1722.755720] Key type lgssc unregistered [ 1723.022218] LNet: 104790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.029849] LNetError: 104790:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.046684] LNet: Removed LNI 192.168.202.133@tcp [ 1723.888174] Key type .llcrypt unregistered [ 1723.889985] Key type ._llcrypt unregistered [ 1740.978920] Key type ._llcrypt registered [ 1740.983619] Key type .llcrypt registered [ 1741.093203] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1742.320490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1742.390149] alg: No test for adler32 (adler32-zlib) [ 1743.516894] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 1743.768607] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1745.480180] Key type lgssc registered [ 1746.906856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1756.232165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1756.262619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1757.859310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1761.981748] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1768.422548] LustreError: 105994:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1768.461034] LustreError: 105994:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1771.502300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1771.903670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1776.029572] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1781.586888] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1785.753519] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1792.570371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1792.837652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1798.262457] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1803.144601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1803.304457] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1812.455737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1812.458931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1812.491199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1813.491776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1813.499049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1813.517345] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1816.650771] LustreError: 108240:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1816.828440] Lustre: server umount lustre-OST0000 complete [ 1823.201364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1823.205071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.228723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1823.235134] Lustre: Skipped 1 previous similar message [ 1826.541821] LustreError: 108443:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1826.552724] LustreError: 108443:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1826.685703] Lustre: server umount lustre-MDT0000 complete [ 1828.835446] LustreError: 105993:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1828.861946] LustreError: 105993:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 1830.192545] LustreError: 105975:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761284997 with bad export cookie 5567891389952947872 [ 1830.216809] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1830.342571] LustreError: 108644:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1830.349334] LustreError: 108644:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1830.677191] Lustre: server umount lustre-MDT0001 complete [ 1841.364390] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1844.288911] Key type lgssc unregistered [ 1844.595108] LNet: 109176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.605898] LNetError: 109176:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.629162] LNet: Removed LNI 192.168.202.133@tcp [ 1845.585184] Key type .llcrypt unregistered [ 1845.587418] Key type ._llcrypt unregistered [ 1862.694820] Key type ._llcrypt registered [ 1862.698746] Key type .llcrypt registered [ 1862.820718] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1864.024950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1864.095876] alg: No test for adler32 (adler32-zlib) [ 1865.199556] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 1865.426560] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1867.120882] Key type lgssc registered [ 1868.138886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1874.987915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1875.000433] LustreError: 110240:0:(tgt_mount.c:2438:server_fill_super()) cfs_race id 716 sleeping [ 1875.005279] Lustre: Skipped 1 previous similar message [ 1875.008736] LustreError: 110233:0:(tgt_mount.c:2438:server_fill_super()) cfs_fail_race id 716 waking [ 1875.015349] LustreError: 110240:0:(tgt_mount.c:2438:server_fill_super()) cfs_fail_race id 716 awake: rc=4978 [ 1875.018353] LustreError: 110240:0:(tgt_mount.c:2450:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1875.032513] LustreError: 110240:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1875.037109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1876.605320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.734780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1885.078517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1889.441590] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1894.785182] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing lsmod [ 1898.158649] LustreError: 111411:0:(tgt_mount.c:2438:server_fill_super()) cfs_race id 716 sleeping [ 1898.243390] LustreError: 111421:0:(tgt_mount.c:2438:server_fill_super()) cfs_fail_race id 716 waking [ 1898.249800] LustreError: 111411:0:(tgt_mount.c:2438:server_fill_super()) cfs_fail_race id 716 awake: rc=4921 [ 1898.268773] LustreError: 111411:0:(tgt_mount.c:2450:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1898.293752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1898.294136] LustreError: 111411:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1898.871370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.922336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1905.641323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1905.656552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.674890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1908.410733] LustreError: 111606:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1908.621562] Lustre: server umount lustre-MDT0000 complete [ 1910.247352] LustreError: 110276:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1910.283763] LustreError: 110276:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 1913.190465] LustreError: 110258:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761285080 with bad export cookie 4326389864684086028 [ 1913.199348] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1913.223705] LustreError: 110258:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1914.466058] LustreError: 110714:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000396c6f74 x1846841213736064/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1914.545919] LustreError: 111810:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1914.548863] LustreError: 111810:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1914.954152] Lustre: server umount lustre-MDT0001 complete [ 1929.498595] LustreError: 112012:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1929.501910] LustreError: 112012:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1929.718257] Lustre: server umount lustre-OST0000 complete [ 1938.344699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.833644] LustreError: 112489:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1938.845346] LustreError: 112489:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 1938.895356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1942.963241] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1944.035674] LustreError: 112489:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1949.157899] LustreError: 112489:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1950.908946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1951.156847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1954.821780] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1960.482518] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1964.451381] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1971.481948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1972.000718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.423306] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1984.323124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 1985.822334] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1994.723710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.729721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1994.748557] Lustre: Skipped 3 previous similar messages [ 1994.751295] Lustre: Skipped 3 previous similar messages [ 1999.559110] LustreError: 114737:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1999.577504] LustreError: 114737:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1999.912630] Lustre: server umount lustre-OST0000 complete [ 2004.969959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.000217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2005.010295] Lustre: Skipped 2 previous similar messages [ 2010.083475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2010.090036] Lustre: Skipped 1 previous similar message [ 2015.208174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2015.212715] Lustre: Skipped 1 previous similar message [ 2018.272217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2018.376707] LustreError: 114939:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2018.385837] LustreError: 114939:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2018.485944] Lustre: server umount lustre-MDT0000 complete [ 2020.325471] LustreError: 112488:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2020.357700] LustreError: 112488:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2022.335597] LustreError: 112469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761285189 with bad export cookie 4326389864684086945 [ 2022.346152] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2022.349383] LustreError: 112469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2022.850431] Lustre: server umount lustre-MDT0001 complete [ 2035.433950] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2038.331906] Key type lgssc unregistered [ 2038.596894] LNet: 115676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.612809] LNetError: 115676:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2039.659316] LNet: Removed LNI 192.168.202.133@tcp [ 2040.534163] Key type .llcrypt unregistered [ 2040.535932] Key type ._llcrypt unregistered [ 2065.856796] Key type ._llcrypt registered [ 2065.858774] Key type .llcrypt registered [ 2065.981870] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2066.905514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2067.047235] alg: No test for adler32 (adler32-zlib) [ 2068.143940] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 2068.348849] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2070.008223] Key type lgssc registered [ 2071.284059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2080.048336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2080.087685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.827333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2085.834234] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2090.979997] LustreError: 117075:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2091.014684] LustreError: 117075:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 2094.118568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2094.534904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2098.266947] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2104.721938] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2108.713980] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2115.787668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2116.177622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.543802] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2126.465207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2127.313250] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2132.444620] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2136.000561] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2141.153497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2141.172373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.190932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2142.182267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2142.187248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2142.194471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.223627] Lustre: Skipped 1 previous similar message [ 2145.724686] LustreError: 119424:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2145.889313] Lustre: server umount lustre-OST0000 complete [ 2150.881320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2150.884422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2150.912239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2155.460074] LustreError: 119630:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2155.468309] LustreError: 119630:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2155.586119] Lustre: server umount lustre-MDT0000 complete [ 2157.540444] LustreError: 118093:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.566687] LustreError: 118093:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2159.823132] LustreError: 117056:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761285327 with bad export cookie 5225112435505717812 [ 2159.832227] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2160.030047] LustreError: 119833:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2160.033946] LustreError: 119833:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2160.358273] Lustre: server umount lustre-MDT0001 complete [ 2171.034415] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2173.768705] Key type lgssc unregistered [ 2174.063872] LNet: 120365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2174.082625] LNetError: 120365:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2175.144878] LNet: Removed LNI 192.168.202.133@tcp [ 2176.007270] Key type .llcrypt unregistered [ 2176.011615] Key type ._llcrypt unregistered [ 2194.418950] Key type ._llcrypt registered [ 2194.420627] Key type .llcrypt registered [ 2194.566696] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2196.051466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2196.097153] alg: No test for adler32 (adler32-zlib) [ 2197.241473] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 2197.637443] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2199.440338] Key type lgssc registered [ 2201.086865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2210.027957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2210.060267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2211.677842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2215.834349] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2216.929576] LustreError: 121570:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.958992] LustreError: 121570:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 2221.044395] LustreError: 121570:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2225.705954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2226.313929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2230.717306] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2236.999532] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2241.199184] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2249.918606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2250.291214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2256.455298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2256.993106] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2263.881197] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2276.838245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2276.845749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2276.865518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2278.489235] LustreError: 123818:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2278.720072] Lustre: server umount lustre-OST0000 complete [ 2287.079381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2287.087553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2287.093828] Lustre: Skipped 2 previous similar messages [ 2287.124738] Lustre: Skipped 2 previous similar messages [ 2288.825130] LustreError: 124020:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2288.847486] LustreError: 124020:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2288.967032] Lustre: server umount lustre-MDT0000 complete [ 2292.194695] LustreError: 121570:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.211202] LustreError: 121570:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2294.144098] LustreError: 121549:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761285461 with bad export cookie 11398258654363867194 [ 2294.156024] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2294.172965] LustreError: 121549:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2294.320525] LustreError: 124222:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2294.323918] LustreError: 124222:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2294.865890] Lustre: server umount lustre-MDT0001 complete [ 2307.303729] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2310.422825] Key type lgssc unregistered [ 2310.775603] LNet: 124754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2310.787506] LNetError: 124754:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2310.809685] LNet: Removed LNI 192.168.202.133@tcp [ 2311.755961] Key type .llcrypt unregistered [ 2311.757789] Key type ._llcrypt unregistered [ 2339.022394] Key type ._llcrypt registered [ 2339.026203] Key type .llcrypt registered [ 2339.236240] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2340.282150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2340.334250] alg: No test for adler32 (adler32-zlib) [ 2341.481566] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 2341.810763] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2343.504184] Key type lgssc registered [ 2344.719737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2352.746692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2352.775913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.367830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.159965] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2363.887255] LustreError: 126153:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.911926] LustreError: 126153:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 2366.297596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2366.692445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2371.158760] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2377.043252] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2380.832171] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2388.340387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2388.785627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2394.818154] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2399.041539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 2399.770711] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2406.926745] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2410.887525] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2410.892213] Lustre: Skipped 2 previous similar messages [ 2415.825357] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2415.843209] Lustre: Skipped 2 previous similar messages [ 2416.254895] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2416.298906] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2422.771889] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.33@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2422.785508] Lustre: Skipped 2 previous similar messages [ 2430.085167] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.33@tcp 192.168.0.[2,10]@tcp [ 2433.912161] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.33/24@tcp in log lustre-MDT0000 [ 2433.916411] Lustre: Skipped 2 previous similar messages [ 2436.099349] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.33/24@tcp [ 2436.104630] Lustre: Skipped 1 previous similar message [ 2442.217947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2442.221315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2442.250671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2442.726367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2442.730496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2443.904199] LustreError: 129610:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2444.359967] Lustre: server umount lustre-OST0000 complete [ 2448.865083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2448.868482] LustreError: Skipped 1 previous similar message [ 2448.879354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.903961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2448.913051] Lustre: Skipped 1 previous similar message [ 2450.402441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.418034] Lustre: Skipped 1 previous similar message [ 2454.933329] LustreError: 129812:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2454.940542] LustreError: 129812:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2455.043695] Lustre: server umount lustre-MDT0000 complete [ 2455.526689] LustreError: 126154:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2455.540038] LustreError: 126154:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2458.990090] LustreError: 126134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761285626 with bad export cookie 3285314734628888808 [ 2458.991348] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2459.029367] LustreError: 126134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2459.196502] LustreError: 130015:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2459.207389] LustreError: 130015:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2459.567217] Lustre: server umount lustre-MDT0001 complete [ 2471.445879] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2474.496599] Key type lgssc unregistered [ 2474.803064] LNet: 130547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2474.827799] LNetError: 130547:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2474.842818] LNet: Removed LNI 192.168.202.133@tcp [ 2475.783350] Key type .llcrypt unregistered [ 2475.789326] Key type ._llcrypt unregistered [ 2505.717280] Key type ._llcrypt registered [ 2505.718836] Key type .llcrypt registered [ 2505.872478] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2507.001921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2507.025261] alg: No test for adler32 (adler32-zlib) [ 2508.239474] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 2508.529738] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2510.288165] Key type lgssc registered [ 2511.900490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2520.171818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2520.211567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2521.762426] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2521.802842] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.33/24@tcp [ 2521.819603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2526.055853] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2531.306916] LustreError: 132105:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.334646] LustreError: 132105:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 2535.529491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2535.966801] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2536.010489] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.33/24@tcp [ 2536.097992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2540.482399] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2546.914892] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2551.053249] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2558.989839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2559.466110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2566.517176] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2566.754650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 2572.365567] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2581.986364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2581.986755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2581.990966] LustreError: Skipped 1 previous similar message [ 2582.009684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2582.015469] Lustre: Skipped 1 previous similar message [ 2587.109421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2587.113328] Lustre: Skipped 1 previous similar message [ 2587.927156] LustreError: 134410:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2588.167196] Lustre: server umount lustre-OST0000 complete [ 2592.227074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.246865] Lustre: Skipped 1 previous similar message [ 2592.262154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2597.346050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2597.363533] Lustre: Skipped 2 previous similar messages [ 2597.849018] LustreError: 134612:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2597.852143] LustreError: 134612:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2597.989475] Lustre: server umount lustre-MDT0000 complete [ 2601.973901] LustreError: 132084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761285769 with bad export cookie 575112444743455697 [ 2601.996155] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2602.011731] LustreError: 132084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2602.208857] LustreError: 134814:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2602.216357] LustreError: 134814:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2602.463325] Lustre: server umount lustre-MDT0001 complete [ 2614.613902] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2617.545650] Key type lgssc unregistered [ 2617.906360] LNet: 135347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2617.916288] LNetError: 135347:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2617.947509] LNet: Removed LNI 192.168.202.133@tcp [ 2618.845972] Key type .llcrypt unregistered [ 2618.847995] Key type ._llcrypt unregistered [ 2646.614603] Key type ._llcrypt registered [ 2646.618937] Key type .llcrypt registered [ 2646.760884] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 2661.349497] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2662.156465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2662.362698] alg: No test for adler32 (adler32-zlib) [ 2663.742414] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 2664.050931] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2665.784182] Key type lgssc registered [ 2667.251856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2677.529901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2684.850455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2691.211679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2699.223157] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2707.740200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2707.818197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2707.835084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2709.216318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2709.259548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2709.423964] Lustre: lustre-MDT0000: new disk, initializing [ 2709.555967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2709.594961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2714.299361] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2724.471165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2724.559061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2724.684311] Lustre: 138728:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2724.757272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2724.760559] Lustre: Skipped 1 previous similar message [ 2724.827556] Lustre: lustre-MDT0001: new disk, initializing [ 2724.959967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2725.020323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2725.031030] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2725.119100] Lustre: 139525:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 2725.130520] Lustre: 139525:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 2725.143141] Lustre: 139525:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 2725.158138] Lustre: 139525:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 2725.168174] Lustre: 139525:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 2725.174042] Lustre: 139525:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 2729.146122] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2736.591494] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2740.591839] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2747.226033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2747.319645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2747.515567] Lustre: lustre-OST0000: new disk, initializing [ 2747.522503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2747.611194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2752.861360] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2755.691888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2755.714869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2755.892506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2760.117328] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2764.689898] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2764.919375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2769.225933] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2769.507819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2776.548498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2776.552190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2776.573496] LustreError: Skipped 1 previous similar message [ 2776.574274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2776.589761] Lustre: Skipped 1 previous similar message [ 2777.647649] LustreError: 141668:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2777.974377] Lustre: server umount lustre-OST0000 complete [ 2786.274208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2786.280078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.292710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2786.298974] Lustre: Skipped 1 previous similar message [ 2788.553866] LustreError: 141871:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2788.558608] LustreError: 141871:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2788.684084] Lustre: server umount lustre-MDT0000 complete [ 2792.820130] LustreError: 138720:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761285960 with bad export cookie 17462982718259550990 [ 2792.822276] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2792.828866] LustreError: 138720:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2792.985508] LustreError: 142072:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2793.023769] LustreError: 142072:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2793.263786] Lustre: server umount lustre-MDT0001 complete [ 2801.860395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2802.375777] LustreError: 142549:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2802.413022] LustreError: 142549:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 2802.476057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.593404] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2807.782759] LustreError: 142549:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2811.878448] LustreError: 142548:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.618846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2817.031805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2821.282972] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2827.438967] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2831.807099] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2839.674930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2840.137179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2846.752109] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2852.687941] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2857.603178] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2857.840443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2861.757423] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2862.042533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2871.057362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2871.140602] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2871.303707] Lustre: lustre-OST0001: new disk, initializing [ 2871.307350] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2871.391940] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2876.805735] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2880.142647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2880.156394] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2880.239116] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2884.141275] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2884.331265] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2887.934210] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2888.159740] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2891.595034] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fc152d1f800.ost_server_uuid 50 [ 2892.864776] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fc152d1f800.ost_server_uuid in FULL state after 0 sec [ 2900.965195] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2900.975412] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2900.984749] Lustre: Skipped 1 previous similar message [ 2900.990659] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2900.993564] Lustre: Skipped 2 previous similar messages [ 2904.895314] LustreError: 146799:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2904.898186] LustreError: 146799:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2904.953771] Lustre: server umount lustre-OST0001 complete [ 2906.085121] LustreError: 144152:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2906.132126] LustreError: 144152:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2909.153776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2909.156367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.172630] LustreError: Skipped 2 previous similar messages [ 2909.190845] Lustre: Skipped 1 previous similar message [ 2909.194559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2909.211690] Lustre: Skipped 1 previous similar message [ 2911.203660] LustreError: 146874:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2911.232602] LustreError: 146874:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2914.231136] LustreError: 147004:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2914.233151] LustreError: 147004:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2914.488961] Lustre: server umount lustre-OST0000 complete [ 2919.399200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2919.402366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.431101] Lustre: Skipped 1 previous similar message [ 2919.439571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2919.444639] Lustre: Skipped 2 previous similar messages [ 2923.630227] LustreError: 147206:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2923.638203] LustreError: 147206:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2923.818366] Lustre: server umount lustre-MDT0000 complete [ 2926.569463] LustreError: 142548:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2926.582615] LustreError: 142548:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2927.210816] LustreError: 142528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761286094 with bad export cookie 17462982718259551914 [ 2927.214104] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2927.221637] LustreError: 142528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2927.599875] Lustre: server umount lustre-MDT0001 complete [ 2945.242061] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2947.728237] Key type lgssc unregistered [ 2947.994339] LNet: 148396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2948.000957] LNetError: 148396:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2948.024789] LNet: Removed LNI 192.168.202.133@tcp [ 2948.839197] Key type .llcrypt unregistered [ 2948.840971] Key type ._llcrypt unregistered [ 2977.346191] Key type ._llcrypt registered [ 2977.348563] Key type .llcrypt registered [ 2977.474740] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 2991.797677] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2992.799565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2992.853990] alg: No test for adler32 (adler32-zlib) [ 2993.982099] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 2994.213704] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2995.872181] Key type lgssc registered [ 2997.030665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3004.066766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3008.676475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3013.710729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3019.213453] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3024.529784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3024.560425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3024.570350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3025.720878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3025.752502] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3025.809230] Lustre: lustre-MDT0000: new disk, initializing [ 3025.888250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3025.905825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3028.581034] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3037.119734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3037.187227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3037.240508] Lustre: 152347:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3037.296698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3037.298790] Lustre: Skipped 1 previous similar message [ 3037.350398] Lustre: lustre-MDT0001: new disk, initializing [ 3037.446410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3037.479096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3037.491084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3037.535781] LustreError: 153144:0:(llog_cat.c:511:llog_cat_current_log()) lustre-MDT0001-osd: next log does not exist, catalog [0x240000400:0x1:0x0] rc=-5 [ 3037.542621] LustreError: 153144:0:(update_trans.c:965:top_trans_stop()) lustre-MDT0001-osd: write updates failed: rc = -5 [ 3037.554422] LustreError: 153144:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 3037.589481] Lustre: 153144:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 514 < left 524, rollback = 2 [ 3037.597054] Lustre: 153144:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/1, destroy: 0/0/0 [ 3037.605187] Lustre: 153144:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 3037.612097] Lustre: 153144:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 3037.619144] Lustre: 153144:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/1, delete: 0/0/0 [ 3037.624064] Lustre: 153144:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 3040.248929] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3045.793108] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3048.373539] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3053.957688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3054.014604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3054.207982] Lustre: lustre-OST0000: new disk, initializing [ 3054.211172] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3054.273295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3058.256479] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3063.892329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3063.895629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3064.067876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3064.157953] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3067.650572] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3067.869660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3071.389427] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3071.599621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3074.529987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3074.548372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3074.551395] Lustre: Skipped 1 previous similar message [ 3074.563559] Lustre: Skipped 1 previous similar message [ 3079.310305] LustreError: 155286:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3079.422927] Lustre: server umount lustre-OST0000 complete [ 3083.747334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3083.767519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3083.783345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3084.770887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3088.737456] LustreError: 155488:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3088.742702] LustreError: 155488:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3088.803969] Lustre: server umount lustre-MDT0000 complete [ 3091.585179] LustreError: 154103:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761286258 with bad export cookie 15356619054972822391 [ 3091.585685] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3091.593737] LustreError: 154103:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3091.694885] LustreError: 155689:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3091.697786] LustreError: 155689:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3091.899941] Lustre: server umount lustre-MDT0001 complete [ 3098.693234] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 02:11:04 (1761286264) [ 3111.810915] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3120.409208] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3128.629338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3133.280518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3137.952368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3142.965585] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3148.594338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3148.631968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3148.839081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3148.867891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3148.955552] Lustre: lustre-MDT0000: new disk, initializing [ 3149.056604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3149.080322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3151.928489] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3160.921050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3160.993670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3161.129430] Lustre: 158680:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3161.180758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3161.183678] Lustre: Skipped 1 previous similar message [ 3161.260729] Lustre: lustre-MDT0001: new disk, initializing [ 3161.363657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3161.394608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3161.403860] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3161.503852] Lustre: 159478:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 3161.513226] Lustre: 159478:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 3161.522601] Lustre: 159478:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 3161.531922] Lustre: 159478:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3161.541507] Lustre: 159478:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 3161.552276] Lustre: 159478:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3161.558118] Lustre: 159478:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 3161.565797] Lustre: 159478:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3161.576393] Lustre: 159478:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 3161.585823] Lustre: 159478:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3161.589803] Lustre: 159478:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 3161.594362] Lustre: 159478:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3164.728211] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3171.156996] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3174.284264] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3179.409404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3179.455632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3179.631337] Lustre: lustre-OST0000: new disk, initializing [ 3179.637619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3179.720048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3184.911943] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3187.305905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3187.332533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3187.451775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3192.045979] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3195.893508] Lustre: Failing over lustre-OST0000 [ 3197.927992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3197.931318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3197.941835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3197.944099] LustreError: Skipped 1 previous similar message [ 3197.949718] Lustre: Skipped 3 previous similar messages [ 3197.985136] LustreError: 161125:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3197.992621] LustreError: 161125:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3198.275565] Lustre: server umount lustre-OST0000 complete [ 3215.973936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3216.324258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3216.349783] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3218.354661] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3220.933249] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3223.863297] Lustre: Failing over lustre-MDT0000 [ 3223.882277] LustreError: 150559:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3224.001905] LustreError: 162104:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3224.022852] LustreError: 162104:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3224.103961] Lustre: server umount lustre-MDT0000 complete [ 3224.568702] LustreError: 158686:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.579098] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 3224.579292] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3224.598411] LustreError: 158686:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 3228.130452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3228.135855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3228.143245] Lustre: Skipped 1 previous similar message [ 3228.149778] LustreError: 158691:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.655128] LustreError: 158686:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.684020] LustreError: 158686:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 3233.763521] LustreError: 158691:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3233.777619] LustreError: 158691:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 3238.882231] LustreError: 158691:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.893240] LustreError: 158691:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 3241.207084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3241.299116] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3241.436670] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3241.526484] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3244.523197] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3246.048770] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3246.567379] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3246.584428] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3246.635714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3255.270849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3255.282485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.300922] Lustre: Skipped 2 previous similar messages [ 3255.316064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3258.812144] LustreError: 162990:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3258.817733] LustreError: 162990:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3258.939694] Lustre: server umount lustre-OST0000 complete [ 3261.923368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3261.924937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3261.928212] Lustre: Skipped 3 previous similar messages [ 3272.163289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3272.166236] Lustre: Skipped 3 previous similar messages [ 3276.257300] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3276.337812] LustreError: 163191:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3276.341623] LustreError: 163191:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3276.447704] Lustre: server umount lustre-MDT0000 complete [ 3277.281459] LustreError: 158688:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3277.297368] LustreError: 158688:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 3279.320983] LustreError: 158672:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761286446 with bad export cookie 15356619054972824603 [ 3279.322170] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3279.337173] LustreError: 158672:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3279.599346] Lustre: server umount lustre-MDT0001 complete [ 3289.253117] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3291.638422] Key type lgssc unregistered [ 3291.871467] LNet: 163925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3291.878558] LNetError: 163925:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3291.901953] LNet: Removed LNI 192.168.202.133@tcp [ 3292.614240] Key type .llcrypt unregistered [ 3292.618380] Key type ._llcrypt unregistered [ 3325.980645] Key type ._llcrypt registered [ 3325.983529] Key type .llcrypt registered [ 3326.071190] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3326.859952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3326.899101] alg: No test for adler32 (adler32-zlib) [ 3327.989153] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 3328.183032] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3329.840320] Key type lgssc registered [ 3330.806152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3337.982588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3338.008091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3339.399710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3342.551918] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3348.726908] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3348.964976] LustreError: 165428:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3348.977170] LustreError: 165428:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 3352.361835] LustreError: 165457:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3352.373704] LustreError: 165457:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3352.378793] LustreError: 165457:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 0, failed: rc = -5 [ 3352.490892] LustreError: 165981:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3352.686909] Lustre: server umount lustre-MDT0000 complete [ 3364.295325] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3372.087630] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3379.621488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3385.256988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3389.596589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3394.593171] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3399.995069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3400.068387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3400.239332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3400.273297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3400.339768] Lustre: lustre-MDT0000: new disk, initializing [ 3400.419479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3400.448885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3404.150130] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3413.296687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3413.374634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3413.431908] Lustre: 168625:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3413.481664] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3413.484333] Lustre: Skipped 1 previous similar message [ 3413.539506] Lustre: lustre-MDT0001: new disk, initializing [ 3413.628965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3413.672771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3413.688890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3413.819822] Lustre: 169423:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 3413.857650] Lustre: 169423:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 3413.875540] Lustre: 169423:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 3413.890756] Lustre: 169423:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 3413.894268] Lustre: 169423:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 3413.903170] Lustre: 169423:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 3417.416836] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3424.080746] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3427.006193] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3431.961796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3432.036784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3432.230389] Lustre: lustre-OST0000: new disk, initializing [ 3432.234132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3432.316917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3436.560617] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3441.235084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3441.246144] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3441.386104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3443.140310] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3446.192706] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3447.432122] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:16:53 (1761286613) [ 3661.792603] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3661.793084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3661.801955] Lustre: Skipped 2 previous similar messages [ 3661.802772] Lustre: Skipped 1 previous similar message [ 3666.231139] LustreError: 172150:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3666.260333] LustreError: 172150:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3666.375339] Lustre: server umount lustre-MDT0000 complete [ 3666.918946] LustreError: 168636:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3666.935022] LustreError: 168636:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 3669.936893] LustreError: 169407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761286837 with bad export cookie 201060322892026703 [ 3669.942432] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3669.955758] LustreError: 169407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3672.034626] LustreError: 172030:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3672.035187] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3672.064157] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3674.082924] LustreError: 169762:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3674.083614] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3679.202676] LustreError: 168633:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3679.202834] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3684.320116] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3684.401666] LustreError: 172352:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3684.405913] LustreError: 172352:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3684.571553] Lustre: server umount lustre-MDT0001 complete [ 3702.240777] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3702.260844] LustreError: 172555:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3702.263608] LustreError: 172555:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3702.389551] Lustre: server umount lustre-OST0000 complete [ 3709.767779] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3717.488383] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3726.230560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3730.707754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3735.735835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3741.512045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3747.866575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3747.927951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3748.103062] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3748.130824] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3748.181930] Lustre: lustre-MDT0000: new disk, initializing [ 3748.270740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3748.292240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3751.446525] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3760.151363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3760.220981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3760.329338] Lustre: 174895:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3760.395745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3760.398527] Lustre: Skipped 1 previous similar message [ 3760.447381] Lustre: lustre-MDT0001: new disk, initializing [ 3760.529201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3760.553509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3760.566140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3760.647060] Lustre: 175691:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 3760.659399] Lustre: 175691:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 3760.665211] Lustre: 175691:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 3760.669094] Lustre: 175691:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3760.679534] Lustre: 175691:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 3760.684731] Lustre: 175691:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3760.695407] Lustre: 175691:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 3760.710832] Lustre: 175691:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3760.713802] Lustre: 175691:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 3760.724822] Lustre: 175691:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3760.731902] Lustre: 175691:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 3760.743531] Lustre: 175691:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3763.747730] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3770.968973] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3774.324186] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3780.349147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3780.417263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3780.593873] Lustre: lustre-OST0000: new disk, initializing [ 3780.602297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3780.678213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3785.030280] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3786.291821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3786.295797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3786.376741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3790.965029] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3794.893360] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3795.124318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3798.499261] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3798.671132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3801.573098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3801.578484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3801.599691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3801.611046] Lustre: Skipped 1 previous similar message [ 3802.088284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3806.540071] LustreError: 177837:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3806.545954] LustreError: 177837:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3806.767722] Lustre: server umount lustre-OST0000 complete [ 3811.813769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3811.821364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3811.835700] Lustre: Skipped 1 previous similar message [ 3811.841788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3811.844673] Lustre: Skipped 2 previous similar messages [ 3824.096192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3824.157598] LustreError: 178039:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3824.160744] LustreError: 178039:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3824.247786] Lustre: server umount lustre-MDT0000 complete [ 3827.319958] LustreError: 174888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761286994 with bad export cookie 201060322892185568 [ 3827.337912] LustreError: 174888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3827.339726] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3827.680169] Lustre: server umount lustre-MDT0001 complete [ 3834.987222] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:23:20 (1761287000) [ 3848.053413] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3856.296174] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3866.480938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3872.999195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3878.763709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3883.821440] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3890.153514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3890.268046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3890.554855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3890.577171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3890.631546] Lustre: lustre-MDT0000: new disk, initializing [ 3890.698837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3890.715748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3894.243380] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3903.174072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3903.253150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3903.325869] Lustre: 181244:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3903.533202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3903.595766] Lustre: 182042:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 3903.602383] Lustre: 182042:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 3903.606089] Lustre: 182042:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 3903.613830] Lustre: 182042:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3903.628015] Lustre: 182042:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 3903.636125] Lustre: 182042:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3903.645749] Lustre: 182042:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 3903.652781] Lustre: 182042:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3903.657614] Lustre: 182042:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 3903.663489] Lustre: 182042:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3903.668222] Lustre: 182042:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 3903.676152] Lustre: 182042:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3906.309112] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3910.831167] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3913.476247] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3918.559918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3918.613784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3922.082125] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3926.053790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3926.068907] Lustre: Skipped 1 previous similar message [ 3926.077262] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3926.141900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3927.829167] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3936.228313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3936.233290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3936.242439] Lustre: Skipped 2 previous similar messages [ 3936.262596] Lustre: Skipped 7 previous similar messages [ 3938.833462] LustreError: 183786:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3938.839358] LustreError: 183786:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3939.065317] Lustre: server umount lustre-OST0000 complete [ 3944.421577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3944.424886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3949.854087] Lustre: server umount lustre-MDT0000 complete [ 3951.592511] LustreError: 181249:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3951.624129] LustreError: 181249:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 3953.265153] LustreError: 181236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287120 with bad export cookie 201060322892186499 [ 3953.277454] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3953.279223] LustreError: 181236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3953.653519] Lustre: server umount lustre-MDT0001 complete [ 3962.723466] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:25:28 (1761287128) [ 3972.993212] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3979.447617] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3986.998180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3992.318931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3997.387722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4002.041677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4008.486262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4008.537641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4008.753654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4008.784395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4008.787276] Lustre: Skipped 3 previous similar messages [ 4008.832448] Lustre: lustre-MDT0000: new disk, initializing [ 4008.834769] Lustre: Skipped 2 previous similar messages [ 4008.933834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4008.937669] Lustre: Skipped 2 previous similar messages [ 4008.953647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4012.219735] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4020.123835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4020.187180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4020.269495] Lustre: 187185:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4020.513500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4020.597573] Lustre: 187985:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 4020.606069] Lustre: 187985:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 4020.610698] Lustre: 187985:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 4020.620863] Lustre: 187985:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4020.631079] Lustre: 187985:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 4020.646219] Lustre: 187985:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4020.650753] Lustre: 187985:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 4020.659126] Lustre: 187985:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4020.675496] Lustre: 187985:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 4020.678608] Lustre: 187985:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4020.684839] Lustre: 187985:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 4020.690807] Lustre: 187985:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4023.633245] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4030.109753] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4033.426530] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4039.567308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4039.609056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4045.036733] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4046.469034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4052.266498] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4060.610281] LustreError: 189730:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4060.613415] LustreError: 189730:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4060.832314] Lustre: server umount lustre-OST0000 complete [ 4066.794663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4066.798300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4066.803655] Lustre: Skipped 1 previous similar message [ 4066.808856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4066.811795] Lustre: Skipped 3 previous similar messages [ 4077.549814] Lustre: 164738:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761287228/real 1761287228] req@00000000e5f48327 x1846842748577920/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761287244 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4078.624162] Lustre: 164738:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761287229/real 1761287229] req@00000000be6352c1 x1846842748578816/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761287245 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4079.072142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4079.239929] Lustre: server umount lustre-MDT0000 complete [ 4082.363035] LustreError: 187176:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287249 with bad export cookie 201060322892187626 [ 4082.365048] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4082.376031] LustreError: 187176:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4082.703424] Lustre: server umount lustre-MDT0001 complete [ 4091.130634] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 4093.403459] Key type lgssc unregistered [ 4093.631699] LNet: 190665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4093.640121] LNetError: 190665:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4094.693982] LNet: Removed LNI 192.168.202.133@tcp [ 4095.280168] Key type .llcrypt unregistered [ 4095.282473] Key type ._llcrypt unregistered [ 4114.877134] Key type ._llcrypt registered [ 4114.878754] Key type .llcrypt registered [ 4115.035020] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4115.629421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4115.636515] alg: No test for adler32 (adler32-zlib) [ 4116.702270] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 4116.885375] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 4118.525232] Key type lgssc registered [ 4119.871707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4128.064883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4128.087627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4129.654684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4133.701259] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4139.004821] LustreError: 192042:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4139.011797] LustreError: 192042:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 4142.858119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4143.240288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4147.374309] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4153.753989] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4156.992236] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4163.089499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4163.452768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4168.143416] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4172.604289] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4174.741258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4194.274665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4194.293089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.309240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4195.302670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4195.305293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.195671] LustreError: 194315:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4200.406834] Lustre: server umount lustre-OST0000 complete [ 4204.513743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4204.536192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4204.546037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4204.550088] Lustre: Skipped 1 previous similar message [ 4210.657496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4210.660228] Lustre: Skipped 2 previous similar messages [ 4215.785650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4215.801370] Lustre: Skipped 1 previous similar message [ 4218.336126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4218.433639] LustreError: 194517:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4218.436219] LustreError: 194517:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4218.503178] Lustre: server umount lustre-MDT0000 complete [ 4219.888862] LustreError: 192039:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4219.903139] LustreError: 192039:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 4221.799548] LustreError: 192022:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287389 with bad export cookie 10722227335074294106 [ 4221.802585] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4221.805564] LustreError: 192022:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4221.940961] LustreError: 194720:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4221.943193] LustreError: 194720:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4222.146921] Lustre: server umount lustre-MDT0001 complete [ 4232.324082] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 4235.346847] Key type lgssc unregistered [ 4235.687795] LNet: 195252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4235.701275] LNetError: 195252:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4236.779197] LNet: Removed LNI 192.168.202.133@tcp [ 4237.626482] Key type .llcrypt unregistered [ 4237.631894] Key type ._llcrypt unregistered [ 4258.925269] Key type ._llcrypt registered [ 4258.926784] Key type .llcrypt registered [ 4259.035828] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4260.133233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4260.194601] alg: No test for adler32 (adler32-zlib) [ 4261.258079] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 4261.498262] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 4263.168208] Key type lgssc registered [ 4264.143797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4270.813605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4270.849797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4272.278493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4275.874415] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4281.828550] LustreError: 196649:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4281.858740] LustreError: 196649:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 4284.917854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4285.278591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4289.659217] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4295.505324] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4299.859634] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4306.592936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4306.875180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4312.175489] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4316.827517] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4317.169795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4324.848140] Lustre: lustre-OST0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 4325.347369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4325.358047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4325.376956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4327.394737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4327.396670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4327.414944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4329.395376] LustreError: 198898:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4329.538132] Lustre: server umount lustre-OST0000 complete [ 4333.839584] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9fc1457e9800.ost_server_uuid 50 [ 4335.326599] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4351.970749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4351.974887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4351.985839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4351.996916] Lustre: Skipped 2 previous similar messages [ 4353.523672] LustreError: 199266:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4353.526229] LustreError: 199266:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4353.611892] Lustre: server umount lustre-MDT0000 complete [ 4356.931205] LustreError: 196630:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287524 with bad export cookie 9629522476855129231 [ 4356.941158] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4356.945730] LustreError: 196630:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4357.062662] LustreError: 199468:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4357.066960] LustreError: 199468:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4357.330410] Lustre: server umount lustre-MDT0001 complete [ 4365.019052] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:32:10 (1761287530) [ 4371.923713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4372.491390] LustreError: 200141:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4372.618554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4376.659179] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4378.083809] LustreError: 200141:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4382.194254] LustreError: 200141:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4384.843493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4385.246408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4389.170177] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4394.662992] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4397.411408] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4402.485873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4402.779462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4407.822436] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4412.305202] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4417.178590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4418.580593] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4418.667899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4418.846729] Lustre: lustre-OST0001: new disk, initializing [ 4418.850360] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4418.945086] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4423.538530] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4427.858571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4427.863401] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4427.932825] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4429.747214] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4432.881303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4432.884333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4432.909371] Lustre: Skipped 1 previous similar message [ 4432.917736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4432.920200] Lustre: Skipped 2 previous similar messages [ 4438.011359] LustreError: 203349:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4438.013738] LustreError: 203349:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4438.049763] Lustre: server umount lustre-OST0000 complete [ 4438.501861] LustreError: 203027:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4438.515723] LustreError: 203027:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 4442.297617] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4442.559108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4445.927985] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4446.177565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4448.749899] LustreError: 201741:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4448.779501] LustreError: 201741:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 4464.098312] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4464.103695] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4464.152131] Lustre: Skipped 1 previous similar message [ 4464.152873] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4464.152877] Lustre: Skipped 3 previous similar messages [ 4465.530542] LustreError: 204073:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4465.553134] LustreError: 204073:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4465.735512] Lustre: server umount lustre-OST0001 complete [ 4475.301771] LustreError: 204274:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4475.305806] LustreError: 204274:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4475.397418] Lustre: server umount lustre-MDT0000 complete [ 4478.586630] LustreError: 200121:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287645 with bad export cookie 9629522476855130302 [ 4478.590763] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4478.616123] LustreError: 200121:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4479.038824] Lustre: server umount lustre-MDT0001 complete [ 4483.748362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4486.682834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4489.837284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4493.486784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4499.947438] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:34:25 (1761287665) [ 4505.660387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4505.869838] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4505.900700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4506.052320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4509.486775] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4516.895840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4517.073975] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4517.135592] Lustre: 205968:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4521.445306] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4526.559412] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4530.872798] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4538.438649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4538.938699] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4546.115506] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4548.782053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 4551.953568] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4560.435083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4560.493746] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4566.101032] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4568.209920] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4572.112548] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4578.308571] Lustre: lustre-OST0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 4578.323353] Lustre: Skipped 5 previous similar messages [ 4578.797703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4578.811108] Lustre: Skipped 4 previous similar messages [ 4581.127062] LustreError: 209171:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4581.137642] LustreError: 209171:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4581.201321] Lustre: server umount lustre-OST0000 complete [ 4583.414918] LustreError: 207600:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4583.424437] LustreError: 207600:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 8 previous similar messages [ 4598.761457] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4598.771236] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4602.678022] Lustre: server umount lustre-OST0001 complete [ 4612.935482] Lustre: server umount lustre-MDT0000 complete [ 4616.278613] LustreError: 205960:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287783 with bad export cookie 9629522476855131702 [ 4616.290628] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4616.331801] LustreError: 205960:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4616.475356] LustreError: 209798:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4616.484933] LustreError: 209798:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4616.784410] Lustre: server umount lustre-MDT0001 complete [ 4621.764735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4625.889647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4630.249159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4634.632615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4641.514321] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:36:46 (1761287806) [ 4656.897258] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 4665.072158] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4675.224083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4682.482165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4690.499514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4700.007357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4709.112075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4709.190772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4709.545973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4709.615085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4709.756825] Lustre: lustre-MDT0000: new disk, initializing [ 4709.957067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4709.968739] Lustre: Skipped 3 previous similar messages [ 4710.019165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4714.376281] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4724.219424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4724.329385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4724.399642] Lustre: 213626:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4724.456493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4724.467043] Lustre: Skipped 1 previous similar message [ 4724.594951] Lustre: lustre-MDT0001: new disk, initializing [ 4724.736651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4724.745718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4724.872200] Lustre: 214423:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 4724.879969] Lustre: 214423:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 4724.888371] Lustre: 214423:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 4724.896461] Lustre: 214423:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 4724.900778] Lustre: 214423:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 4724.906019] Lustre: 214423:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 4728.726767] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4736.645939] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4740.376912] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4747.168465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4747.246357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4747.523958] Lustre: lustre-OST0000: new disk, initializing [ 4747.533360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4752.986731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4753.002205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4753.168690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4753.770969] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4761.329150] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4766.120629] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4766.354267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4770.468989] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4770.815708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4773.856986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4773.863484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4773.866305] LustreError: Skipped 3 previous similar messages [ 4773.867331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4773.867335] Lustre: Skipped 7 previous similar messages [ 4773.884368] Lustre: Skipped 4 previous similar messages [ 4787.168224] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4787.238832] LustreError: 216569:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4787.241589] LustreError: 216569:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4787.483210] Lustre: server umount lustre-OST0000 complete [ 4797.969308] Lustre: server umount lustre-MDT0000 complete [ 4799.459170] LustreError: 213635:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4799.477053] LustreError: 213635:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 13 previous similar messages [ 4801.617545] LustreError: 213618:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761287968 with bad export cookie 9629522476855133487 [ 4801.623743] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4801.637539] LustreError: 213618:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4802.178151] Lustre: server umount lustre-MDT0001 complete [ 4809.974919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4814.007407] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4821.602402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4825.638376] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4831.410578] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4835.229272] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4841.506860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4841.823443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4841.834702] Lustre: Skipped 4 previous similar messages [ 4847.540528] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4853.178248] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4856.991607] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4857.209852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4860.940149] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4861.233795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4865.510087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4865.512421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4865.520465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4865.530836] Lustre: Skipped 3 previous similar messages [ 4865.539321] Lustre: Skipped 7 previous similar messages [ 4877.280169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4877.589826] Lustre: server umount lustre-OST0000 complete [ 4883.581990] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4887.420745] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4915.792246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4920.705643] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4925.640549] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4931.444684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4931.451458] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4931.452056] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4931.458958] Lustre: Skipped 1 previous similar message [ 4934.730518] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec [ 4941.799431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4941.809307] LustreError: Skipped 1 previous similar message [ 4943.960414] LustreError: 221839:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4943.967254] LustreError: 221839:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 4944.150917] Lustre: server umount lustre-OST0000 complete [ 4953.550046] Lustre: server umount lustre-MDT0000 complete [ 4956.836750] LustreError: 220195:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288124 with bad export cookie 9629522476855134404 [ 4956.844028] LustreError: 220195:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4956.844035] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4957.247756] Lustre: server umount lustre-MDT0001 complete [ 4964.410810] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:42:10 (1761288130) [ 4970.624919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4971.135679] LustreError: 222912:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4971.158512] LustreError: 222912:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 4974.724666] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4982.413684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4986.018942] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4990.605304] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4993.708743] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4999.118122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5004.271793] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5009.217596] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5012.654319] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5016.062857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5019.623881] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5019.816878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5025.707147] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5025.807739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5025.929804] Lustre: lustre-OST0001: new disk, initializing [ 5025.932048] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5030.769635] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5033.042632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5033.054666] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5033.114376] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5037.494826] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5040.221293] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5040.357196] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5042.867634] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5042.999512] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5048.807972] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5048.816929] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5048.821157] Lustre: Skipped 5 previous similar messages [ 5048.827340] Lustre: Skipped 9 previous similar messages [ 5050.210643] Lustre: server umount lustre-OST0001 complete [ 5053.762958] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5053.921797] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5056.205272] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5056.333101] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5083.036683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5087.182695] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5088.747645] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5088.756740] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5088.758962] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5088.774177] Lustre: Skipped 1 previous similar message [ 5090.565428] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5091.499443] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5093.857502] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5093.866595] LustreError: Skipped 4 previous similar messages [ 5107.168155] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5107.222717] Lustre: server umount lustre-OST0001 complete [ 5108.196533] LustreError: 224509:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5108.207972] LustreError: 224509:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 20 previous similar messages [ 5126.471985] Lustre: server umount lustre-OST0000 complete [ 5134.620388] Lustre: server umount lustre-MDT0000 complete [ 5136.607306] LustreError: 222892:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288303 with bad export cookie 9629522476855135377 [ 5136.609966] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5136.612727] LustreError: 222892:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5142.838666] Lustre: server umount lustre-MDT0001 complete [ 5145.630274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5147.567700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5149.495903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5151.391496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5154.592498] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:45:20 (1761288320) [ 5157.768710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5157.922227] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5157.939371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5158.036360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5158.039563] Lustre: Skipped 6 previous similar messages [ 5159.738378] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5163.454281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5163.519937] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5163.540480] Lustre: 230916:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5165.453105] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5168.119336] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5169.966945] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5173.603947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5173.705627] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5176.621727] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5179.139461] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5184.055864] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5184.101410] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5186.901117] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5189.331576] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5190.071509] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5191.914787] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5193.048888] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5195.135748] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5195.249946] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5197.314698] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fc1467e9800.ost_server_uuid 50 [ 5198.203897] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fc1467e9800.ost_server_uuid in FULL state after 0 sec [ 5198.704873] Lustre: Permanently deactivating lustre-OST0001 [ 5198.713787] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5198.795547] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5200.286054] Lustre: Permanently reactivating lustre-OST0001 [ 5208.244305] LustreError: 234920:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5208.250624] LustreError: 234920:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 5208.283489] Lustre: server umount lustre-OST0001 complete [ 5216.427024] Lustre: server umount lustre-OST0000 complete [ 5224.431479] LustreError: 234924:0:(osp_dev.c:692:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 5228.650961] Lustre: server umount lustre-MDT0000 complete [ 5232.878269] LustreError: 230907:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288400 with bad export cookie 9629522476855136665 [ 5232.889838] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5232.900630] LustreError: 230907:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5233.472243] Lustre: server umount lustre-MDT0001 complete [ 5239.765904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5245.195948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5250.779820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5255.400926] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5263.668612] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:47:08 (1761288428) [ 5265.364855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5272.265361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5272.643173] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5272.667320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5272.675241] Lustre: Skipped 5 previous similar messages [ 5275.848957] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5283.344613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5283.410208] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5283.432025] Lustre: 237081:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5286.858354] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5291.037155] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5294.121550] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5299.205318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5299.259365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5299.386854] Lustre: Permanently deactivating lustre-OST0000 [ 5299.399820] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5303.799755] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5305.356786] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5305.371500] Lustre: Skipped 1 previous similar message [ 5308.656920] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5314.250533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5320.016768] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5324.347736] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5324.897550] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5331.012636] Lustre: Permanently reactivating lustre-OST0000 [ 5331.027781] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5331.031967] Lustre: Skipped 2 previous similar messages [ 5334.557642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5334.564062] Lustre: Skipped 11 previous similar messages [ 5334.577485] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5334.585412] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.612672] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5334.665218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.683816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5340.140899] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5340.159792] Lustre: Skipped 24 previous similar messages [ 5340.274927] Lustre: server umount lustre-OST0001 complete [ 5349.769438] Lustre: server umount lustre-OST0000 complete [ 5355.504903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5355.510534] LustreError: Skipped 5 previous similar messages [ 5359.488019] Lustre: server umount lustre-MDT0000 complete [ 5362.673308] LustreError: 237073:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288529 with bad export cookie 9629522476855138373 [ 5362.679211] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5362.709848] LustreError: 237073:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5363.042334] Lustre: server umount lustre-MDT0001 complete [ 5372.695718] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5375.291041] Key type lgssc unregistered [ 5375.519434] LNet: 241477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5375.529944] LNetError: 241477:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5375.546412] LNet: Removed LNI 192.168.202.133@tcp [ 5376.357512] Key type .llcrypt unregistered [ 5376.359264] Key type ._llcrypt unregistered [ 5397.733792] Key type ._llcrypt registered [ 5397.738750] Key type .llcrypt registered [ 5397.849752] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5398.442503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5398.532691] alg: No test for adler32 (adler32-zlib) [ 5399.521664] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 5399.684885] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5401.305589] Key type lgssc registered [ 5402.323323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5405.253837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5410.825060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5410.850353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5412.244512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5415.572706] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5422.563431] LustreError: 242927:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5422.583716] LustreError: 242927:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 5422.844318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5422.943874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5423.116168] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5423.173900] Lustre: 242917:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5423.193583] Lustre: Permanently deactivating lustre-MDT0001 [ 5423.214920] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5423.467124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5426.866633] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5427.396708] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5427.406296] LustreError: 242958:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5427.410929] LustreError: 242958:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5427.426701] LustreError: 242958:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 5431.175591] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5433.951746] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5439.095888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5439.343125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5443.526154] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5447.739681] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5448.553795] LustreError: 244550:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5453.599362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5453.768423] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5456.822464] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5456.822903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5456.828027] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5457.830648] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5462.358024] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5466.540261] Lustre: Permanently reactivating lustre-MDT0001 [ 5466.571561] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5466.590961] Lustre: Skipped 2 previous similar messages [ 5469.631416] Lustre: lustre-MDT0001: Client ed09ef69-9696-4010-8d21-ae3a8ab32648 (at 192.168.202.33@tcp) reconnecting [ 5473.200254] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5474.847395] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5479.835846] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5480.027287] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5483.766874] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5483.946486] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5484.989903] Lustre: 242922:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 5484.994474] Lustre: 242922:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5484.999618] Lustre: 242922:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 5485.005274] Lustre: 242922:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 5485.009964] Lustre: 242922:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5485.014297] Lustre: 242922:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5486.332089] Lustre: Permanently deactivating lustre-MDT0001 [ 5486.349577] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5486.352198] Lustre: Skipped 2 previous similar messages [ 5494.384514] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5497.887616] Lustre: 242922:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 506 < left 979, rollback = 2 [ 5497.896582] Lustre: 242922:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 5497.906169] Lustre: 242922:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 2/8/6, destroy: 0/0/0 [ 5497.910041] Lustre: 242922:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5497.923285] Lustre: 242922:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 6/979/0 [ 5497.934901] Lustre: 242922:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5497.942589] Lustre: 242922:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 5497.958503] Lustre: 242922:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5497.963275] Lustre: 242922:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 8/130/4, delete: 0/0/0 [ 5497.968257] Lustre: 242922:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5497.974133] Lustre: 242922:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0 [ 5497.978514] Lustre: 242922:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 5503.332606] LustreError: 247017:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5503.430991] Lustre: server umount lustre-MDT0000 complete [ 5503.971091] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5503.976075] LustreError: 244964:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5503.996315] Lustre: Skipped 1 previous similar message [ 5506.326408] LustreError: 246849:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288673 with bad export cookie 17492714156038915017 [ 5506.329311] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5506.343525] LustreError: 246849:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5506.497925] LustreError: 247218:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5506.499981] LustreError: 247218:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5506.728685] Lustre: server umount lustre-MDT0001 complete [ 5509.970218] LustreError: 247418:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5509.980034] LustreError: 247418:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5510.019598] Lustre: server umount lustre-OST0000 complete [ 5520.211485] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:51:25 (1761288685) [ 5529.078643] LustreError: 248423:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5529.083747] LustreError: 248423:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5529.327676] Lustre: server umount lustre-OST0001 complete [ 5534.308929] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 5542.513701] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5549.819809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5555.494159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5561.100308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5566.053685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5571.802613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5571.861830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5572.032049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5572.036363] Lustre: Skipped 2 previous similar messages [ 5572.063205] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5572.119545] Lustre: lustre-MDT0000: new disk, initializing [ 5572.219081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5572.237616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5575.314880] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5582.918543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5582.988514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5583.085874] Lustre: 250609:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5583.131318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5583.135026] Lustre: Skipped 1 previous similar message [ 5583.184313] Lustre: lustre-MDT0001: new disk, initializing [ 5583.253631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5583.284395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5583.290306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5583.353323] Lustre: 251407:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 5583.360027] Lustre: 251407:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 3 previous similar messages [ 5583.363980] Lustre: 251407:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5583.370196] Lustre: 251407:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 5583.373378] Lustre: 251407:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 5583.378629] Lustre: 251407:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 5583.384790] Lustre: 251407:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 5583.392318] Lustre: 251407:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 5583.402772] Lustre: 251407:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5583.412175] Lustre: 251407:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 5583.416358] Lustre: 251407:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5583.423955] Lustre: 251407:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 5585.996782] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5590.985970] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5593.221429] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5597.531769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5597.577452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5597.728401] Lustre: lustre-OST0000: new disk, initializing [ 5597.732059] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5601.002574] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5605.398119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5605.403278] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5605.471531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5605.661229] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5607.028545] LustreError: 251769:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5612.132874] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5612.170738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5612.239435] Lustre: lustre-OST0001: new disk, initializing [ 5612.242329] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5612.282633] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5612.286205] Lustre: Skipped 1 previous similar message [ 5614.965279] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5617.121448] LustreError: 251769:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5617.138000] LustreError: 251769:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5619.007322] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5619.806811] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5621.781050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5621.785773] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5621.834750] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5627.232281] LustreError: 251769:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5629.412414] Lustre: lustre-OST0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 5631.975249] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5631.975584] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5631.981079] Lustre: Skipped 1 previous similar message [ 5631.986314] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5631.989465] Lustre: Skipped 1 previous similar message [ 5634.164119] LustreError: 254107:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5634.170187] LustreError: 254107:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5634.197205] Lustre: server umount lustre-OST0001 complete [ 5634.527108] LustreError: 252928:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5634.541955] LustreError: 252928:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 5642.212772] LustreError: 254021:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5642.219479] LustreError: 254021:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 5651.425569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5651.425598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5651.429485] LustreError: Skipped 2 previous similar messages [ 5651.430851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5651.434959] Lustre: Skipped 1 previous similar message [ 5651.441620] Lustre: Skipped 1 previous similar message [ 5652.452847] LustreError: 254021:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5652.459216] LustreError: 254021:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 5653.427415] LustreError: 254331:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5653.430662] LustreError: 254331:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5653.511196] Lustre: server umount lustre-OST0000 complete [ 5657.571128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5657.572250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5657.577394] Lustre: Skipped 1 previous similar message [ 5657.580243] Lustre: Skipped 2 previous similar messages [ 5661.435901] Lustre: server umount lustre-MDT0000 complete [ 5663.241065] LustreError: 251390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288830 with bad export cookie 17492714156038917614 [ 5663.244630] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5663.246387] LustreError: 251390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5669.510078] Lustre: server umount lustre-MDT0001 complete [ 5674.561270] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5675.903250] Key type lgssc unregistered [ 5676.057359] LNet: 255267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5676.062922] LNetError: 255267:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5677.095165] LNet: Removed LNI 192.168.202.133@tcp [ 5677.508298] Key type .llcrypt unregistered [ 5677.509931] Key type ._llcrypt unregistered [ 5692.567249] Key type ._llcrypt registered [ 5692.568890] Key type .llcrypt registered [ 5692.625938] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 5700.995334] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5701.405312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5701.464077] alg: No test for adler32 (adler32-zlib) [ 5702.378558] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 5702.498293] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5704.105376] Key type lgssc registered [ 5704.781616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5710.267684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5714.153968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5717.648767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5721.959495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5726.546890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5726.581699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5726.595948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5727.753983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5727.787478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5727.841786] Lustre: lustre-MDT0000: new disk, initializing [ 5727.914783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5727.925285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5730.230440] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5735.662497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5735.694945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5735.739859] Lustre: 259206:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5735.771267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5735.774029] Lustre: Skipped 1 previous similar message [ 5735.813174] Lustre: lustre-MDT0001: new disk, initializing [ 5735.858313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5735.875831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5735.880814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5735.924578] Lustre: 260006:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 5735.929611] Lustre: 260006:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5735.933753] Lustre: 260006:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 5735.941450] Lustre: 260006:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 5735.947073] Lustre: 260006:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5735.952030] Lustre: 260006:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5737.549449] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5741.295892] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5742.966095] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5746.154770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5746.195124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5746.318061] Lustre: lustre-OST0000: new disk, initializing [ 5746.321146] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5746.365171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5748.923729] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5751.818141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5751.822985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5751.869136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5752.811110] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5754.714022] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5754.811866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5756.520492] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5756.625721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5762.018669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5762.020780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5762.021991] LustreError: Skipped 1 previous similar message [ 5762.022828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5762.028199] Lustre: Skipped 1 previous similar message [ 5762.035943] Lustre: Skipped 1 previous similar message [ 5763.574256] LustreError: 262148:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5763.668651] Lustre: server umount lustre-OST0000 complete [ 5766.625178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5766.628738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5766.635374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5771.596210] LustreError: 262349:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5771.599204] LustreError: 262349:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5771.646539] Lustre: server umount lustre-MDT0000 complete [ 5773.579871] LustreError: 259199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761288940 with bad export cookie 4385110350022398416 [ 5773.582537] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5773.586547] LustreError: 259199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5773.662598] LustreError: 262550:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5773.666661] LustreError: 262550:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5773.789730] Lustre: server umount lustre-MDT0001 complete [ 5778.806167] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 02:55:44 (1761288944) [ 5782.086511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5782.320870] LustreError: 263221:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5782.330503] LustreError: 263221:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 5782.352583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5784.106441] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5787.617700] LustreError: 263221:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5787.711539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5787.869614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5789.543512] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5792.091085] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5793.937739] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5797.262367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5797.430144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5800.122204] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5802.816072] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5848.546605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5848.547415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5848.549217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5848.549224] Lustre: Skipped 2 previous similar messages [ 5848.560567] Lustre: Skipped 2 previous similar messages [ 5853.667847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5853.670970] Lustre: Skipped 2 previous similar messages [ 5854.131172] LustreError: 265547:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5854.134255] LustreError: 265547:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5854.225608] Lustre: server umount lustre-OST0000 complete [ 5856.217474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5861.639368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5861.809834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5864.373699] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5867.033333] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5878.184154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.190828] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.194144] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5878.201024] Lustre: Skipped 1 previous similar message [ 5903.843694] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5903.844114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5903.847127] LustreError: Skipped 1 previous similar message [ 5903.855034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5903.859399] Lustre: Skipped 1 previous similar message [ 5907.448406] LustreError: 267213:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5907.452915] LustreError: 267213:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5907.566331] Lustre: server umount lustre-OST0000 complete [ 5911.009031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5911.012577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5911.017893] Lustre: Skipped 1 previous similar message [ 5914.082573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5914.085751] Lustre: Skipped 2 previous similar messages [ 5915.592501] LustreError: 267415:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5915.596448] LustreError: 267415:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5915.653401] Lustre: server umount lustre-MDT0000 complete [ 5917.837539] LustreError: 263201:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289085 with bad export cookie 4385110350022399326 [ 5917.843176] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5917.845322] LustreError: 263201:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5918.068210] Lustre: server umount lustre-MDT0001 complete [ 5923.774724] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5925.182203] Key type lgssc unregistered [ 5925.360788] LNet: 268147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5925.365202] LNetError: 268147:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5925.376731] LNet: Removed LNI 192.168.202.133@tcp [ 5925.810071] Key type .llcrypt unregistered [ 5925.811701] Key type ._llcrypt unregistered [ 5938.774151] Key type ._llcrypt registered [ 5938.775888] Key type .llcrypt registered [ 5938.828211] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5939.292200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5939.335887] alg: No test for adler32 (adler32-zlib) [ 5940.230675] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 5940.358605] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5941.961694] Key type lgssc registered [ 5942.619147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5947.365721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5947.377335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5948.722940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5950.888083] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5955.916896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5956.116306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5958.278285] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5961.527166] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5963.638615] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5967.326806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5967.500282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5970.343707] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5973.235329] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5973.673718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5996.514296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5996.516188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5996.519441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5996.524986] Lustre: Skipped 1 previous similar message [ 6001.210175] LustreError: 272562:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6001.294477] Lustre: server umount lustre-OST0000 complete [ 6006.753870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.756223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6006.759019] Lustre: Skipped 1 previous similar message [ 6006.762074] Lustre: Skipped 1 previous similar message [ 6009.362356] LustreError: 272764:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6009.368491] LustreError: 272764:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6009.445773] Lustre: server umount lustre-MDT0000 complete [ 6011.874469] LustreError: 269523:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6011.891026] LustreError: 269523:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 6011.904691] LustreError: 269503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289179 with bad export cookie 8689922144472365172 [ 6011.909417] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6011.909888] LustreError: 269503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6011.996468] LustreError: 272966:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6012.000377] LustreError: 272966:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6012.131860] Lustre: server umount lustre-MDT0001 complete [ 6017.422296] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6018.753847] Key type lgssc unregistered [ 6018.926085] LNet: 273497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6018.931992] LNetError: 273497:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6018.943361] LNet: Removed LNI 192.168.202.133@tcp [ 6019.326627] Key type .llcrypt unregistered [ 6019.328611] Key type ._llcrypt unregistered [ 6030.317791] Key type ._llcrypt registered [ 6030.319464] Key type .llcrypt registered [ 6030.384714] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6031.007435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6031.026461] alg: No test for adler32 (adler32-zlib) [ 6031.937755] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 6032.064624] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6033.674198] Key type lgssc registered [ 6034.290667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6038.515767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6038.525922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6039.803482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6041.647606] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6045.375500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6045.531908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6047.174845] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6049.734338] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6051.521394] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6055.086912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6055.289618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6057.974640] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6060.433103] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6064.488639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6071.778877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6071.782679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6071.789933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6074.849776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6074.851219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6074.855900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.465480] LustreError: 277071:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6076.546222] Lustre: server umount lustre-OST0000 complete [ 6079.970228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6079.971802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6079.975474] Lustre: Skipped 1 previous similar message [ 6079.980509] Lustre: Skipped 2 previous similar messages [ 6084.232692] LustreError: 277275:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6084.236530] LustreError: 277275:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6084.289332] Lustre: server umount lustre-MDT0000 complete [ 6085.092030] LustreError: 275811:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6085.104512] LustreError: 275811:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 6085.933405] LustreError: 274661:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289253 with bad export cookie 6379667156116543953 [ 6085.936547] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6085.938704] LustreError: 274661:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6086.022441] LustreError: 277476:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6086.024678] LustreError: 277476:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6086.133680] Lustre: server umount lustre-MDT0001 complete [ 6090.433262] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6091.596821] Key type lgssc unregistered [ 6091.755701] LNet: 278009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6091.759486] LNetError: 278009:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6091.772470] LNet: Removed LNI 192.168.202.133@tcp [ 6092.178676] Key type .llcrypt unregistered [ 6092.180374] Key type ._llcrypt unregistered [ 6103.707832] Key type ._llcrypt registered [ 6103.709357] Key type .llcrypt registered [ 6103.761439] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6104.323043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6104.361641] alg: No test for adler32 (adler32-zlib) [ 6105.304972] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 6105.424987] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6107.032232] Key type lgssc registered [ 6107.647449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6111.578154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6111.586828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6112.862671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6114.786729] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6118.985458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6119.171252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6120.870141] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6123.222234] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6124.818050] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6127.771123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6127.951806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6130.538186] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6132.885373] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6138.220069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6148.579899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6148.582302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6148.586417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6148.592075] Lustre: Skipped 1 previous similar message [ 6148.597135] Lustre: Skipped 1 previous similar message [ 6153.700090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6153.703511] Lustre: Skipped 1 previous similar message [ 6154.489218] LustreError: 282414:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6154.627671] Lustre: server umount lustre-OST0000 complete [ 6158.818323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6158.819200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6158.830054] Lustre: Skipped 1 previous similar message [ 6158.838375] Lustre: Skipped 1 previous similar message [ 6162.755930] LustreError: 282615:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6162.758684] LustreError: 282615:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6162.808592] Lustre: server umount lustre-MDT0000 complete [ 6163.937997] LustreError: 279395:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6163.945179] LustreError: 279395:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 6164.573495] LustreError: 280112:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289331 with bad export cookie 14986732270086041365 [ 6164.576577] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6164.582655] LustreError: 280112:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6164.660781] LustreError: 282817:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6164.666340] LustreError: 282817:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6164.804926] Lustre: server umount lustre-MDT0001 complete [ 6169.276609] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6170.668586] Key type lgssc unregistered [ 6170.840692] LNet: 283348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6170.847298] LNetError: 283348:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6170.859844] LNet: Removed LNI 192.168.202.133@tcp [ 6171.267398] Key type .llcrypt unregistered [ 6171.269396] Key type ._llcrypt unregistered [ 6180.760733] Key type ._llcrypt registered [ 6180.762336] Key type .llcrypt registered [ 6180.822301] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6181.374645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6181.437122] alg: No test for adler32 (adler32-zlib) [ 6182.374162] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 6182.504754] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6184.112216] Key type lgssc registered [ 6184.703683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6188.853470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6188.872737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6190.165980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6192.199728] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6196.453922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6196.657432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6198.449188] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6201.080580] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6202.712946] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6205.986122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6206.179404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6208.714555] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6211.366585] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6215.400453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6226.915079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.915983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6226.921213] Lustre: Skipped 1 previous similar message [ 6226.927593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6228.211115] LustreError: 286978:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6228.287981] Lustre: server umount lustre-OST0000 complete [ 6232.034580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6232.035498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6232.040706] Lustre: Skipped 1 previous similar message [ 6232.043268] Lustre: Skipped 1 previous similar message [ 6237.153352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6237.156153] Lustre: Skipped 2 previous similar messages [ 6242.273314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6242.276175] Lustre: Skipped 1 previous similar message [ 6244.320388] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6244.375429] LustreError: 287179:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6244.378973] LustreError: 287179:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6244.426931] Lustre: server umount lustre-MDT0000 complete [ 6246.046454] LustreError: 285287:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289413 with bad export cookie 920485403131383459 [ 6246.050172] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6246.051395] LustreError: 285287:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6246.138127] LustreError: 287381:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6246.141182] LustreError: 287381:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6246.254623] Lustre: server umount lustre-MDT0001 complete [ 6250.901298] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6252.171622] Key type lgssc unregistered [ 6252.321542] LNet: 287913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6252.325149] LNetError: 287913:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6252.332523] LNet: Removed LNI 192.168.202.133@tcp [ 6252.695233] Key type .llcrypt unregistered [ 6252.696726] Key type ._llcrypt unregistered [ 6257.657389] Key type ._llcrypt registered [ 6257.659064] Key type .llcrypt registered [ 6257.711422] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6265.308618] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 6272.915056] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6273.369060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6273.431105] alg: No test for adler32 (adler32-zlib) [ 6274.324546] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 6274.433072] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6276.024227] Key type lgssc registered [ 6276.503860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6280.525602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6283.552099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6286.613831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6289.628120] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6292.955682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6292.981368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6292.988558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6294.093839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6294.114783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6294.156746] Lustre: lustre-MDT0000: new disk, initializing [ 6294.205447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6294.215822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6295.902105] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6301.134204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6301.168883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6301.212533] Lustre: 291396:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6301.246497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6301.249062] Lustre: Skipped 1 previous similar message [ 6301.294622] Lustre: lustre-MDT0001: new disk, initializing [ 6301.345605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6301.365415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6301.370460] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6301.420628] Lustre: 292196:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6301.426689] Lustre: 292196:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6301.430183] Lustre: 292196:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6301.434384] Lustre: 292196:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6301.438217] Lustre: 292196:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6301.441706] Lustre: 292196:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6303.199464] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6307.107706] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6308.803334] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6311.834552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6311.865625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6311.980330] Lustre: lustre-OST0000: new disk, initializing [ 6311.983212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6312.024824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6314.437291] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6318.049043] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6319.689540] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6320.137102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6320.141047] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6320.188388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6320.811430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6322.419481] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6322.523533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6325.219672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6325.220399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6325.226439] LustreError: Skipped 1 previous similar message [ 6325.233626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6325.238834] Lustre: Skipped 1 previous similar message [ 6329.403642] LustreError: 294342:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6329.510750] Lustre: server umount lustre-OST0000 complete [ 6332.386194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6332.389519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6332.395458] Lustre: Skipped 1 previous similar message [ 6332.399561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6335.457447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6335.457843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6337.287360] LustreError: 294543:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6337.289738] LustreError: 294543:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6337.333238] Lustre: server umount lustre-MDT0000 complete [ 6338.997889] LustreError: 291388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289506 with bad export cookie 2610956084002560270 [ 6338.998470] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6339.002699] LustreError: 291388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6339.073786] LustreError: 294746:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6339.075910] LustreError: 294746:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6339.193864] Lustre: server umount lustre-MDT0001 complete [ 6342.869458] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:05:09 (1761289509) [ 6345.839330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6346.094225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6347.831381] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6351.329684] LustreError: 295416:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6351.336952] LustreError: 295416:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 6351.922341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6352.103711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6353.775967] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6356.336463] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6357.940173] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6360.907368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6361.040105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6363.368486] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6365.576311] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6381.539464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6381.539943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6381.547911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6381.550504] Lustre: Skipped 2 previous similar messages [ 6385.265133] LustreError: 297687:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6385.270031] LustreError: 297687:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6385.356456] Lustre: server umount lustre-OST0000 complete [ 6388.193929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6388.198375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6388.205501] Lustre: Skipped 1 previous similar message [ 6388.208584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6393.217056] LustreError: 297889:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6393.219744] LustreError: 297889:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6393.249186] Lustre: server umount lustre-MDT0000 complete [ 6394.648920] LustreError: 295397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289561 with bad export cookie 2610956084002561180 [ 6394.650548] LustreError: 296137:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 6394.652394] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6394.653256] LustreError: 295397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6394.796078] Lustre: server umount lustre-MDT0001 complete [ 6398.966653] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6400.171576] Key type lgssc unregistered [ 6400.311748] LNet: 298623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6400.315482] LNetError: 298623:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6400.324579] LNet: Removed LNI 192.168.202.133@tcp [ 6400.671794] Key type .llcrypt unregistered [ 6400.673370] Key type ._llcrypt unregistered [ 6410.983236] Key type ._llcrypt registered [ 6410.985082] Key type .llcrypt registered [ 6411.039440] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 6417.703220] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6418.190742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6418.231630] alg: No test for adler32 (adler32-zlib) [ 6419.128354] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 6419.243761] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6420.840220] Key type lgssc registered [ 6421.348432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6424.818046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6427.247479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6429.534517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6432.228206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6434.274194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6436.382821] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6439.240719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6439.262732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6439.269590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6440.372800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6440.388649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6440.429834] Lustre: lustre-MDT0000: new disk, initializing [ 6440.467276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6440.477224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6442.149738] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6447.135352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6447.165515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6447.211347] Lustre: 302399:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6447.250432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6447.253289] Lustre: Skipped 1 previous similar message [ 6447.297830] Lustre: lustre-MDT0001: new disk, initializing [ 6447.346031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6447.363712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6447.368306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6447.402821] Lustre: 303195:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6447.408848] Lustre: 303195:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6447.413506] Lustre: 303195:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6447.416853] Lustre: 303195:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6447.422597] Lustre: 303195:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6447.428046] Lustre: 303195:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6448.909342] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6452.425684] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6453.975326] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6456.691836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6456.724552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6456.850712] Lustre: lustre-OST2710: new disk, initializing [ 6456.853636] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6456.904707] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6459.465488] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6459.902952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6459.908058] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6459.959655] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6462.852792] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 6465.504099] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6465.533449] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6465.597895] Lustre: lustre-OST03e8: new disk, initializing [ 6465.600317] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6465.635536] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6466.309193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6466.313432] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6466.349567] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6467.924966] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6470.178703] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6473.380871] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6473.498798] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6475.078205] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6475.171029] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6476.782458] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6476.869374] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6478.393227] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6478.489224] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6481.891616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6481.896905] Lustre: Skipped 3 previous similar messages [ 6481.896936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6481.901657] Lustre: Skipped 3 previous similar messages [ 6485.822587] LustreError: 306738:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6485.852563] Lustre: server umount lustre-MDT0000 complete [ 6487.274716] LustreError: 303178:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289654 with bad export cookie 7163318069749570473 [ 6487.279424] LustreError: 303178:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6487.280855] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6487.348037] LustreError: 306941:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6487.350740] LustreError: 306941:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6487.451060] Lustre: server umount lustre-MDT0001 complete [ 6499.321167] LustreError: 307142:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6499.324941] LustreError: 307142:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6499.357511] Lustre: server umount lustre-OST2710 complete [ 6511.155348] LustreError: 307344:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6511.157742] LustreError: 307344:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6511.240848] Lustre: server umount lustre-OST03e8 complete [ 6517.949996] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 6521.648108] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6525.630144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6528.177458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6530.650097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6533.069294] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6535.730162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6535.753160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.851605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6535.868072] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6535.907849] Lustre: lustre-MDT0000: new disk, initializing [ 6535.936206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6535.944822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6537.475292] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6542.270377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6542.302248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6542.341787] Lustre: 310136:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6542.466761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6542.506102] Lustre: 310938:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6542.513382] Lustre: 310938:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 6542.516917] Lustre: 310938:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6542.523804] Lustre: 310938:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6542.529288] Lustre: 310938:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6542.533044] Lustre: 310938:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6542.536037] Lustre: 310938:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6542.539256] Lustre: 310938:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6542.542402] Lustre: 310938:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6542.544609] Lustre: 310938:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6542.547143] Lustre: 310938:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6542.550468] Lustre: 310938:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6544.103250] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6547.770169] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6549.269833] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6551.872143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6551.894777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6551.990299] Lustre: lustre-OST0000: new disk, initializing [ 6551.991804] Lustre: Skipped 1 previous similar message [ 6551.993866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6551.997121] Lustre: Skipped 2 previous similar messages [ 6552.026432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6552.029262] Lustre: Skipped 1 previous similar message [ 6554.103405] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6555.133276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6555.136153] Lustre: Skipped 1 previous similar message [ 6555.138524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6555.166339] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6557.527260] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6559.222366] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6559.314352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6560.947045] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6561.049653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6565.347430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6565.350598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6565.357289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6565.360088] Lustre: Skipped 1 previous similar message [ 6570.466578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6570.468772] Lustre: Skipped 1 previous similar message [ 6575.586647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6575.591881] Lustre: Skipped 1 previous similar message [ 6576.096197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6576.116459] LustreError: 313079:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6576.122510] LustreError: 313079:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6576.210087] Lustre: server umount lustre-OST0000 complete [ 6578.658567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6578.663643] LustreError: Skipped 1 previous similar message [ 6578.665791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6578.676547] Lustre: Skipped 1 previous similar message [ 6580.708227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6580.710893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6580.723341] Lustre: Skipped 1 previous similar message [ 6584.596634] LustreError: 313282:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6584.610823] LustreError: 313282:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6584.861889] Lustre: server umount lustre-MDT0000 complete [ 6585.826531] LustreError: 310149:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6585.854078] LustreError: 310149:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 6 previous similar messages [ 6588.844397] LustreError: 310929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289756 with bad export cookie 7163318069749572181 [ 6588.854745] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6589.479987] Lustre: server umount lustre-MDT0001 complete [ 6598.438828] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:09:23 (1761289763) [ 6600.006145] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6602.165404] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:09:27 (1761289767) [ 6605.445830] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6612.881292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6616.857424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6621.282962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6625.479889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6626.592782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6633.681751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6633.909564] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6633.934104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6634.121824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6637.017806] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6642.497304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6642.575521] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6642.602891] Lustre: 315616:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6644.930737] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6648.238496] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6650.672266] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6654.719986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6654.777663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6654.886169] Lustre: Denying initial registration attempt from nid 192.168.202.133@tcp, specified as failover [ 6654.891267] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6654.896504] LustreError: 317235:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -99 [ 6654.901864] LustreError: 317235:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 6654.906074] LustreError: 317235:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6654.974958] Lustre: server umount lustre-OST0000 complete [ 6654.979994] LustreError: 317235:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6661.092101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6661.100793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6661.103327] Lustre: Skipped 2 previous similar messages [ 6665.751121] LustreError: 317603:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6665.754217] LustreError: 317603:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6665.819975] Lustre: server umount lustre-MDT0000 complete [ 6666.212053] LustreError: 315625:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6666.221106] LustreError: 315625:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 6668.264201] LustreError: 315608:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289835 with bad export cookie 7163318069749573105 [ 6668.267733] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6668.270461] LustreError: 315608:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6668.518672] Lustre: server umount lustre-MDT0001 complete [ 6675.247613] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6676.782509] Key type lgssc unregistered [ 6676.979739] LNet: 318337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6676.983275] LNetError: 318337:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6676.991831] LNet: Removed LNI 192.168.202.133@tcp [ 6677.484718] Key type .llcrypt unregistered [ 6677.486727] Key type ._llcrypt unregistered [ 6683.837411] Key type ._llcrypt registered [ 6683.838692] Key type .llcrypt registered [ 6683.916746] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6684.556712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6684.586506] alg: No test for adler32 (adler32-zlib) [ 6685.573554] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 6685.730875] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6687.360197] Key type lgssc registered [ 6688.071086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6692.116236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6694.946768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6697.289805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6699.846863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6700.623705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6711.375310] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6715.986681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6715.995674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6717.139514] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6717.156215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6717.254073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6719.286708] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6724.167040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6724.224147] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6724.241305] Lustre: 321070:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6724.412977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6726.593364] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6729.791877] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6731.543662] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6735.059448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6735.107519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6735.220048] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6735.320060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6738.292766] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6741.248684] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6746.085995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6746.087446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6746.090498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6746.095530] Lustre: Skipped 1 previous similar message [ 6746.102491] Lustre: Skipped 1 previous similar message [ 6748.467120] LustreError: 323355:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6748.550262] Lustre: server umount lustre-OST0000 complete [ 6751.202173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6751.205125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6751.208238] Lustre: Skipped 1 previous similar message [ 6751.211831] Lustre: Skipped 1 previous similar message [ 6756.324767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6756.334473] Lustre: Skipped 1 previous similar message [ 6756.811577] LustreError: 323558:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6756.814239] LustreError: 323558:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6756.890627] Lustre: server umount lustre-MDT0000 complete [ 6759.441472] LustreError: 321061:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761289926 with bad export cookie 15977410718859426536 [ 6759.447938] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6759.550581] LustreError: 323760:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6759.553364] LustreError: 323760:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6759.747508] Lustre: server umount lustre-MDT0001 complete [ 6767.194855] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6768.796543] Key type lgssc unregistered [ 6768.978309] LNet: 324291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6768.985827] LNetError: 324291:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6769.002737] LNet: Removed LNI 192.168.202.133@tcp [ 6769.563755] Key type .llcrypt unregistered [ 6769.565391] Key type ._llcrypt unregistered [ 6786.403226] Key type ._llcrypt registered [ 6786.405017] Key type .llcrypt registered [ 6786.526107] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6787.353133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6787.367716] alg: No test for adler32 (adler32-zlib) [ 6788.391987] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 6788.528644] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6790.176180] Key type lgssc registered [ 6790.918627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6796.319557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6796.331176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6797.675042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6800.664369] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6807.011361] LustreError: 325667:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.034803] LustreError: 325667:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 6807.337454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6807.625800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6810.273415] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6813.660415] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6815.916304] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6821.066739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6821.348108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6825.003988] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6828.848374] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6841.845267] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 6842.850329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6842.854623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6842.871407] Lustre: Skipped 2 previous similar messages [ 6842.880088] Lustre: Skipped 1 previous similar message [ 6843.894997] LustreError: 327937:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6843.951583] Lustre: server umount lustre-MDT0000 complete [ 6846.700858] LustreError: 325647:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290013 with bad export cookie 7661865752735976352 [ 6846.711670] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6846.943051] LustreError: 326779:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6846.964826] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 6846.969058] Lustre: Skipped 1 previous similar message [ 6847.971202] LustreError: 326779:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6847.976304] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6852.005530] LustreError: 325662:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6852.005605] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6852.015979] Lustre: Skipped 1 previous similar message [ 6852.855516] LustreError: 328139:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6852.862642] LustreError: 328139:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6853.033319] Lustre: server umount lustre-MDT0001 complete [ 6858.024468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6864.064648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6871.522079] LustreError: 324977:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005de3df7d x1846846375880832/t0(0) o250->MGC192.168.202.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6871.764863] LustreError: 327274:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6871.771858] LustreError: 327274:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 6871.839803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6871.879810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 6874.205513] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6879.030332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6879.229640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6881.163569] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6881.318106] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6884.049251] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6885.027512] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6887.237927] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6888.299457] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6891.491464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6891.500645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6891.503326] Lustre: Skipped 2 previous similar messages [ 6901.732099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6901.735759] Lustre: Skipped 3 previous similar messages [ 6905.312122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6905.344324] LustreError: 330442:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6905.352880] LustreError: 330442:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6905.433217] Lustre: server umount lustre-OST0000 complete [ 6909.921295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6909.926680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6909.938328] Lustre: Skipped 1 previous similar message [ 6914.518258] LustreError: 330647:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6914.523685] LustreError: 330647:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6914.611291] Lustre: server umount lustre-MDT0000 complete [ 6917.042736] LustreError: 325646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290084 with bad export cookie 7661865752735983492 [ 6917.045541] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6917.055734] LustreError: 325646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6917.384428] Lustre: server umount lustre-MDT0001 complete [ 6924.512929] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6926.265177] Key type lgssc unregistered [ 6926.486637] LNet: 331381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6926.501463] LNetError: 331381:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6926.511723] LNet: Removed LNI 192.168.202.133@tcp [ 6927.072103] Key type .llcrypt unregistered [ 6927.073614] Key type ._llcrypt unregistered [ 6941.975177] Key type ._llcrypt registered [ 6941.976781] Key type .llcrypt registered [ 6942.048490] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6942.694101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6942.790247] alg: No test for adler32 (adler32-zlib) [ 6943.733215] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 6943.883753] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6945.512195] Key type lgssc registered [ 6946.240772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6951.427154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6951.441467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6952.723685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6955.011234] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6959.565556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6959.773733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6961.901922] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6964.870807] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6966.787352] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6969.315325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6969.316039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6969.321086] Lustre: Skipped 1 previous similar message [ 6974.429110] LustreError: 334150:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6974.437434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6974.449153] Lustre: Skipped 2 previous similar messages [ 6974.498370] Lustre: server umount lustre-MDT0000 complete [ 6976.525941] LustreError: 332739:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290143 with bad export cookie 9720978780069792495 [ 6976.529756] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6976.534095] LustreError: 332739:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6976.612263] LustreError: 334353:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6976.616024] LustreError: 334353:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6976.732571] Lustre: server umount lustre-MDT0001 complete [ 6984.411369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6984.555261] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6984.576931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6984.708237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6986.761553] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6991.532231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6991.591393] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6991.610426] Lustre: 335123:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6991.752670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6994.125884] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6997.657648] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6999.918570] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7004.738554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7004.888363] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7004.897824] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7004.903709] LustreError: 336784:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -2 [ 7004.909466] LustreError: 336784:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 7004.914104] LustreError: 336784:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 7004.994057] Lustre: server umount lustre-OST0000 complete [ 7004.996042] LustreError: 336784:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7010.472674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7010.590009] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7010.725491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7014.282290] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7017.565941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 7017.918793] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7023.150790] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7023.207901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7023.257348] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7023.334765] Lustre: lustre-OST0001: new disk, initializing [ 7023.338282] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7023.396087] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7026.961211] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7030.729323] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7030.823368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7030.827991] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7030.898152] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7035.875677] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7035.879768] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7035.887064] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7035.891024] Lustre: Skipped 1 previous similar message [ 7038.902127] LustreError: 338908:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7038.911233] LustreError: 338908:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7038.969971] Lustre: server umount lustre-OST0001 complete [ 7041.011128] LustreError: 337252:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7041.018679] LustreError: 337252:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 7043.044850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7043.049231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7043.055450] Lustre: Skipped 1 previous similar message [ 7043.063765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7044.074115] LustreError: 337252:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7044.079337] LustreError: 337252:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 7047.607404] LustreError: 339111:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7047.611093] LustreError: 339111:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7047.709240] Lustre: server umount lustre-OST0000 complete [ 7053.282368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7053.286899] LustreError: Skipped 1 previous similar message [ 7053.288902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7053.294160] Lustre: Skipped 1 previous similar message [ 7053.301291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7053.304040] Lustre: Skipped 3 previous similar messages [ 7056.405736] LustreError: 339312:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7056.409134] LustreError: 339312:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7056.455301] Lustre: server umount lustre-MDT0000 complete [ 7058.781183] LustreError: 335114:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290225 with bad export cookie 9720978780069793034 [ 7058.782614] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7058.792649] LustreError: 335114:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7059.030551] Lustre: server umount lustre-MDT0001 complete [ 7065.912559] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 7067.433284] Key type lgssc unregistered [ 7067.622994] LNet: 340045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7067.628606] LNetError: 340045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7067.639532] LNet: Removed LNI 192.168.202.133@tcp [ 7068.111904] Key type .llcrypt unregistered [ 7068.114294] Key type ._llcrypt unregistered [ 7086.756101] Key type ._llcrypt registered [ 7086.757730] Key type .llcrypt registered [ 7086.863769] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 7096.650464] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7097.204566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7097.222656] alg: No test for adler32 (adler32-zlib) [ 7098.156187] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 7098.277456] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 7099.889562] Key type lgssc registered [ 7100.504971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7105.534892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7109.714578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7113.756795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7117.926313] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7122.577994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7122.614138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7122.629975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7123.793767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7123.824156] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7123.878831] Lustre: lustre-MDT0000: new disk, initializing [ 7123.944718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7123.960289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7126.501150] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7133.493913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7133.530934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7133.598219] Lustre: 343970:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7133.647577] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7133.651046] Lustre: Skipped 1 previous similar message [ 7133.651129] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7133.697565] Lustre: lustre-MDT0001: new disk, initializing [ 7133.757160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7133.781698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7133.792415] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7133.834677] Lustre: 344771:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 508 < left 524, rollback = 2 [ 7133.842815] Lustre: 344771:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/5, destroy: 0/0/0 [ 7133.851202] Lustre: 344771:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 7133.858721] Lustre: 344771:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 7133.864591] Lustre: 344771:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 7133.868346] Lustre: 344771:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 7135.919478] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7140.746682] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7142.693247] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7147.043382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7147.075585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7147.244754] Lustre: lustre-OST0000: new disk, initializing [ 7147.249275] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7147.308302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7151.065182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7151.078042] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7151.153629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7151.329220] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7156.536720] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7159.095658] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7159.232218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7161.732485] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7161.858786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7166.435545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7166.436221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7166.439835] LustreError: Skipped 1 previous similar message [ 7166.447871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7166.452071] Lustre: Skipped 1 previous similar message [ 7169.141136] LustreError: 346907:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7169.296891] Lustre: server umount lustre-OST0000 complete [ 7174.626145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7174.629508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7174.639516] Lustre: Skipped 1 previous similar message [ 7174.642506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7176.673767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7176.675240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7181.793473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7181.796584] Lustre: Skipped 2 previous similar messages [ 7186.400149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7186.460390] LustreError: 347108:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7186.464349] LustreError: 347108:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7186.558879] Lustre: server umount lustre-MDT0000 complete [ 7189.099615] LustreError: 343961:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290356 with bad export cookie 14664640454470899054 [ 7189.101634] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7189.104830] LustreError: 343961:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7189.197197] LustreError: 347312:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7189.199961] LustreError: 347312:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7189.326912] Lustre: server umount lustre-MDT0001 complete [ 7194.603741] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 03:19:20 (1761290360) [ 7198.530250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7202.342077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7216.287473] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 7222.820388] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7228.737695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7233.120483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7237.520122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7242.178045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7246.967317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7247.017301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7247.180055] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7247.213836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7247.305217] Lustre: lustre-MDT0000: new disk, initializing [ 7247.385326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7247.409204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7249.973068] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7257.497237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7257.559888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7257.610808] Lustre: 351087:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7257.642701] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7257.645685] Lustre: Skipped 1 previous similar message [ 7257.691967] Lustre: lustre-MDT0001: new disk, initializing [ 7257.751061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7257.774975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7257.779645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7257.830502] Lustre: 351883:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 7257.836791] Lustre: 351883:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 7257.842378] Lustre: 351883:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 7257.846709] Lustre: 351883:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7257.853943] Lustre: 351883:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 7257.859558] Lustre: 351883:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7257.866311] Lustre: 351883:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 7257.874524] Lustre: 351883:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7257.879709] Lustre: 351883:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 7257.886301] Lustre: 351883:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7257.890298] Lustre: 351883:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 7257.894457] Lustre: 351883:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7260.278100] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7265.866686] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7268.418466] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7272.710667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7272.751311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7272.904571] Lustre: lustre-OST0000: new disk, initializing [ 7272.907579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7272.952916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7276.082947] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7279.642028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7279.646203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7279.712294] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7280.882849] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7283.284514] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7283.423644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7285.497807] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7285.624770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7289.828263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7289.829988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7289.835949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7289.843320] Lustre: Skipped 3 previous similar messages [ 7292.738509] LustreError: 354028:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7292.742305] LustreError: 354028:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7292.867140] Lustre: server umount lustre-OST0000 complete [ 7299.045478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7299.048530] LustreError: Skipped 1 previous similar message [ 7299.050541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7299.056568] Lustre: Skipped 1 previous similar message [ 7301.408686] LustreError: 354230:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7301.411980] LustreError: 354230:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7301.487605] Lustre: server umount lustre-MDT0000 complete [ 7304.071878] LustreError: 351079:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290471 with bad export cookie 14664640454470899978 [ 7304.077075] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7304.081629] LustreError: 351079:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7304.333455] Lustre: server umount lustre-MDT0001 complete [ 7310.124588] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:21:16 (1761290476) [ 7314.796696] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:21:20 (1761290480) [ 7320.609952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7321.018502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7323.800395] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7326.178759] LustreError: 355446:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7326.191360] LustreError: 355446:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 7329.553891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7332.287905] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7335.492957] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7337.582400] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7341.434526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7345.209650] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7348.776489] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7357.707744] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7358.616209] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7359.544296] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7360.434773] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7361.525478] Lustre: Failing over lustre-MDT0000 [ 7361.613980] LustreError: 357887:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7361.616757] LustreError: 357887:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7363.736665] Lustre: server umount lustre-MDT0000 complete [ 7364.581434] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7364.583663] LustreError: 355442:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7364.588120] Lustre: Skipped 3 previous similar messages [ 7364.596737] LustreError: 355442:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 7365.607164] LustreError: 357606:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7369.698913] LustreError: 356160:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7369.706051] LustreError: 356160:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 7374.819072] LustreError: 356160:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7374.825969] LustreError: 356160:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 7379.609765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7379.719107] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7379.884591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7379.887889] Lustre: Skipped 2 previous similar messages [ 7379.918245] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7380.961090] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7382.403807] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7385.067946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7385.079847] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7385.099662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7388.154941] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7389.339963] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7392.314479] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7393.842234] Lustre: Failing over lustre-MDT0000 [ 7393.974435] LustreError: 358969:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7393.978628] LustreError: 358969:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7394.035555] Lustre: server umount lustre-MDT0000 complete [ 7395.298638] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7395.300979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7395.302742] LustreError: 355446:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7395.302755] LustreError: 355446:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 7395.315280] Lustre: Skipped 1 previous similar message [ 7410.177955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7410.290314] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7410.518301] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7411.674086] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7413.229797] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7415.793834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7415.800077] Lustre: Skipped 2 previous similar messages [ 7415.813713] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7415.848756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7418.327548] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7419.297762] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7426.021028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7426.022149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7426.023979] Lustre: Skipped 5 previous similar messages [ 7429.073664] LustreError: 360000:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7429.076839] LustreError: 360000:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7429.131745] Lustre: server umount lustre-MDT0000 complete [ 7430.955277] LustreError: 355429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290598 with bad export cookie 14664640454470902715 [ 7430.955832] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7430.959626] LustreError: 355429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7431.150157] Lustre: server umount lustre-MDT0001 complete [ 7443.541188] Lustre: server umount lustre-OST0000 complete [ 7452.030183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7452.259458] LustreError: 361228:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7452.270047] LustreError: 361228:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 11 previous similar messages [ 7452.301319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7452.307034] Lustre: Skipped 1 previous similar message [ 7454.308366] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7458.922373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7460.933420] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7463.758199] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7465.528432] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7468.917975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7471.855641] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7474.787900] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7477.287133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7482.341266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7482.342156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7482.347260] Lustre: Skipped 4 previous similar messages [ 7482.352095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7482.357024] Lustre: Skipped 1 previous similar message [ 7483.967832] Lustre: server umount lustre-OST0000 complete [ 7487.838586] Lustre: server umount lustre-MDT0000 complete [ 7489.732678] LustreError: 363679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290656 with bad export cookie 14664640454470903450 [ 7489.735087] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7489.739884] LustreError: 363679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7489.915553] Lustre: server umount lustre-MDT0001 complete [ 7494.817204] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 7496.073279] Key type lgssc unregistered [ 7496.244358] LNet: 364411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7496.249195] LNetError: 364411:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7496.260908] LNet: Removed LNI 192.168.202.133@tcp [ 7496.632177] Key type .llcrypt unregistered [ 7496.634069] Key type ._llcrypt unregistered [ 7511.097517] Key type ._llcrypt registered [ 7511.099154] Key type .llcrypt registered [ 7511.144662] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7511.651813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7511.662376] alg: No test for adler32 (adler32-zlib) [ 7512.564757] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 7512.687064] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 7514.288238] Key type lgssc registered [ 7514.782100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7518.370359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7518.383144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7519.627812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7521.437835] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7525.643307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7525.822189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7527.568230] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7530.338780] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7532.221433] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7535.316260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7535.474656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7538.123558] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7540.798244] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7544.685576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7556.066554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7556.069579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7556.074947] Lustre: Skipped 2 previous similar messages [ 7556.080502] Lustre: Skipped 2 previous similar messages [ 7561.186608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7561.189954] Lustre: Skipped 2 previous similar messages [ 7561.803375] LustreError: 368153:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7561.865795] Lustre: server umount lustre-MDT0000 complete [ 7563.699442] LustreError: 365867:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290730 with bad export cookie 16346066026387285470 [ 7563.701751] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7563.706883] LustreError: 365867:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7563.780121] LustreError: 368356:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7563.783134] LustreError: 368356:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7563.890594] Lustre: server umount lustre-MDT0001 complete [ 7574.838810] LustreError: 368558:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7574.841638] LustreError: 368558:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7574.921974] Lustre: server umount lustre-OST0000 complete [ 7581.092609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7581.324750] LustreError: 369288:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7581.336244] LustreError: 369288:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 7581.368171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7583.144897] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7586.788762] LustreError: 369288:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7587.810120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7588.047334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7590.151948] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7593.027345] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7594.975873] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7598.725193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7598.901627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7601.875966] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7604.741312] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7606.063693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7618.531709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7618.532577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7618.538460] Lustre: Skipped 2 previous similar messages [ 7618.541302] Lustre: Skipped 1 previous similar message [ 7623.651670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7623.655978] Lustre: Skipped 3 previous similar messages [ 7623.943053] LustreError: 371639:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7623.946156] LustreError: 371639:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7623.989748] Lustre: server umount lustre-MDT0000 complete [ 7625.798591] LustreError: 369268:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290793 with bad export cookie 16346066026387286569 [ 7625.803572] LustreError: 369268:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7625.803956] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7625.988715] Lustre: server umount lustre-MDT0001 complete [ 7638.004787] LustreError: 372042:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7638.007044] LustreError: 372042:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7638.107611] Lustre: server umount lustre-OST0000 complete [ 7644.820683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7645.033993] LustreError: 372869:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7645.067877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7646.835077] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7650.273945] LustreError: 372869:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7650.700057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7652.556200] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7654.949253] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7656.623670] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7660.075297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7663.042435] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7665.766930] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7670.506525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7675.878314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7675.880471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7675.882640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7675.888964] Lustre: Skipped 1 previous similar message [ 7675.894124] Lustre: Skipped 1 previous similar message [ 7678.900268] LustreError: 375116:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7678.903655] LustreError: 375116:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7678.985936] Lustre: server umount lustre-OST0000 complete [ 7680.993888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7680.999476] Lustre: Skipped 1 previous similar message [ 7686.114363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7686.117426] Lustre: Skipped 3 previous similar messages [ 7695.328135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7695.375569] LustreError: 375317:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7695.378614] LustreError: 375317:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7695.412151] Lustre: server umount lustre-MDT0000 complete [ 7696.353359] LustreError: 372887:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7696.360511] LustreError: 372887:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 7697.002652] LustreError: 372849:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761290864 with bad export cookie 16346066026387287577 [ 7697.004676] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7697.007966] LustreError: 372849:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7697.181783] Lustre: server umount lustre-MDT0001 complete [ 7702.186963] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 7703.645255] Key type lgssc unregistered [ 7703.822166] LNet: 376051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7703.827862] LNetError: 376051:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7703.842690] LNet: Removed LNI 192.168.202.133@tcp [ 7704.271557] Key type .llcrypt unregistered [ 7704.273153] Key type ._llcrypt unregistered [ 7717.544147] Key type ._llcrypt registered [ 7717.545745] Key type .llcrypt registered [ 7717.596781] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7718.007940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7718.043258] alg: No test for adler32 (adler32-zlib) [ 7718.940133] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 7719.051705] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 7720.656283] Key type lgssc registered [ 7721.152209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7724.743663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7724.752669] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7724.757100] LustreError: 377449:0:(osd_handler.c:8747:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7724.769950] LustreError: 377449:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7724.773794] LustreError: 377449:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7724.777671] LustreError: 377449:0:(tgt_mount.c:2450:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7724.781309] LustreError: 377449:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7728.037858] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7728.041789] LustreError: 377848:0:(osd_handler.c:8747:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7728.056792] LustreError: 377848:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7728.060453] LustreError: 377848:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7728.064248] LustreError: 377848:0:(tgt_mount.c:2450:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7728.067547] LustreError: 377848:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7736.348249] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 7737.586716] Key type lgssc unregistered [ 7737.731592] LNet: 378831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7737.735225] LNetError: 378831:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7737.745401] LNet: Removed LNI 192.168.202.133@tcp [ 7738.097571] Key type .llcrypt unregistered [ 7738.099234] Key type ._llcrypt unregistered [ 7747.294575] Key type ._llcrypt registered [ 7747.295895] Key type .llcrypt registered [ 7747.347413] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 7755.443151] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7755.980349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7755.996576] alg: No test for adler32 (adler32-zlib) [ 7756.884759] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 7756.990540] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 7758.592213] Key type lgssc registered [ 7759.062268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7762.914358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7766.446359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7770.456993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7778.111856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7786.466362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7786.530647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7786.539495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7787.828942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7787.855244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7787.933683] Lustre: lustre-MDT0000: new disk, initializing [ 7788.019355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7788.042728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7792.368516] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7804.905212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7805.037208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7805.196596] Lustre: 381939:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7805.346747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7805.370211] Lustre: Skipped 1 previous similar message [ 7805.597606] Lustre: lustre-MDT0001: new disk, initializing [ 7805.930402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7806.032444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7806.069624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7806.215063] Lustre: 382736:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 7806.235666] Lustre: 382736:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 7806.238021] Lustre: 382736:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 7806.251574] Lustre: 382736:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 7806.269971] Lustre: 382736:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 7806.290053] Lustre: 382736:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 7813.674560] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7827.486047] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7835.408975] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7843.554799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7843.636185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7843.928546] Lustre: lustre-OST0000: new disk, initializing [ 7843.936055] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7844.048852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7850.141195] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7851.118952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7851.135652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7851.266640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7857.323527] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7860.815764] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7861.016412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7864.362466] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7864.538626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7866.849777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7866.859885] Lustre: Skipped 1 previous similar message [ 7866.860622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7866.861146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7866.884316] LustreError: Skipped 1 previous similar message [ 7871.972898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7871.983695] Lustre: Skipped 2 previous similar messages [ 7872.136401] LustreError: 384881:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7872.319757] Lustre: server umount lustre-OST0000 complete [ 7877.089516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7877.109694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7877.115873] Lustre: Skipped 1 previous similar message [ 7881.633865] LustreError: 385082:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7881.639168] LustreError: 385082:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7881.725765] Lustre: server umount lustre-MDT0000 complete [ 7884.763182] LustreError: 381931:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761291051 with bad export cookie 16155570022562809712 [ 7884.767562] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7884.783243] LustreError: 381931:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7884.982606] LustreError: 385284:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7884.994785] LustreError: 385284:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7885.380353] Lustre: server umount lustre-MDT0001 complete [ 7891.857477] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7893.001740] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:30:58 (1761291058) [ 7898.086993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7898.497749] LustreError: 386004:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7898.518429] LustreError: 386004:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 7898.549416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7901.952907] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7903.718373] LustreError: 386004:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7907.811870] LustreError: 386004:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7908.821976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7909.066971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7912.091875] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7916.739533] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7919.982759] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7925.778295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7926.076294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7931.505632] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7936.715618] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7944.050903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7944.126873] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7944.216657] Lustre: lustre-OST0001: new disk, initializing [ 7944.219802] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7944.280933] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7949.608692] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7950.993867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7950.998904] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7951.144051] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7957.025311] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7961.576751] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7961.580788] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7961.586209] Lustre: Skipped 2 previous similar messages [ 7961.586794] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7961.592788] Lustre: Skipped 1 previous similar message [ 7966.019192] LustreError: 389203:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7966.030546] LustreError: 389203:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7966.125820] Lustre: server umount lustre-OST0001 complete [ 7966.696434] LustreError: 387604:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7966.707329] LustreError: 387604:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 7971.825703] LustreError: 387602:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7971.834438] LustreError: 387602:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 7982.057305] LustreError: 387604:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7982.062970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7982.090206] LustreError: 387604:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 7982.097625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7982.103094] Lustre: Skipped 2 previous similar messages [ 7987.704182] LustreError: 389427:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7987.713636] LustreError: 389427:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7987.885678] Lustre: server umount lustre-OST0000 complete [ 7991.266370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7991.269241] LustreError: Skipped 1 previous similar message [ 7991.273130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7991.283910] Lustre: Skipped 1 previous similar message [ 7991.287368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7991.290485] Lustre: Skipped 2 previous similar messages [ 7997.339213] LustreError: 389629:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7997.343294] LustreError: 389629:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7997.402545] Lustre: server umount lustre-MDT0000 complete [ 8000.251338] LustreError: 385983:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761291167 with bad export cookie 16155570022562810636 [ 8000.258741] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8000.274962] LustreError: 385983:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8006.703417] Lustre: server umount lustre-MDT0001 complete [ 8016.606628] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 8018.856951] Key type lgssc unregistered [ 8019.099947] LNet: 390364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8019.104807] LNetError: 390364:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8019.118837] LNet: Removed LNI 192.168.202.133@tcp [ 8019.717501] Key type .llcrypt unregistered [ 8019.720070] Key type ._llcrypt unregistered [ 8048.350975] Key type ._llcrypt registered [ 8048.352601] Key type .llcrypt registered [ 8048.426351] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 8061.609135] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 8062.384127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8062.514653] alg: No test for adler32 (adler32-zlib) [ 8063.594537] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 8063.785991] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 8065.482451] Key type lgssc registered [ 8066.501485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8074.967888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8081.294489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8088.064589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8094.963101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8102.500258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8102.571690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8102.593701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8103.812224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8103.850814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8103.922623] Lustre: lustre-MDT0000: new disk, initializing [ 8104.016244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8104.044989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8107.980857] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8118.204613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8118.336200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8118.523790] Lustre: 394313:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8118.664077] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8118.668706] Lustre: Skipped 1 previous similar message [ 8118.732804] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8118.773382] Lustre: lustre-MDT0001: new disk, initializing [ 8118.851803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8118.885016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8118.901257] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8122.991770] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8129.829837] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8133.273917] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8139.837094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8139.895407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8140.106160] Lustre: lustre-OST0000: new disk, initializing [ 8140.119799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8140.262563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8145.704801] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8148.016233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8148.029176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8148.120163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8152.437240] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8156.645034] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8156.936943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8161.043732] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8161.264291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8163.300543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8163.311826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8163.335131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8163.813298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8163.814605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8168.934219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8168.940707] Lustre: Skipped 2 previous similar messages [ 8169.275091] LustreError: 397253:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8169.492089] Lustre: server umount lustre-OST0000 complete [ 8174.054566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8174.059307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8174.066063] Lustre: Skipped 1 previous similar message [ 8174.070849] Lustre: Skipped 1 previous similar message [ 8178.362429] LustreError: 397454:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8178.365541] LustreError: 397454:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8178.451153] Lustre: server umount lustre-MDT0000 complete [ 8181.540065] LustreError: 394305:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761291348 with bad export cookie 13396319341756376936 [ 8181.545188] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8181.552808] LustreError: 394305:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8181.682731] LustreError: 397656:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8181.695327] LustreError: 397656:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8181.883765] Lustre: server umount lustre-MDT0001 complete [ 8188.533456] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 03:35:54 (1761291354) [ 8194.480381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8199.278276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8199.561781] Lustre: lustre-MDT0000: new disk, initializing [ 8199.581496] LustreError: 398924:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8199.596299] LustreError: 398924:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 8199.637572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8202.540559] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8204.775748] LustreError: 398924:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8208.788799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8208.872728] LustreError: 398924:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8209.022269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8211.946749] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8216.465500] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8219.702963] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8221.411963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8221.416664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8221.436710] Lustre: Skipped 1 previous similar message [ 8221.448099] Lustre: Skipped 1 previous similar message [ 8227.655513] LustreError: 400270:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8227.677145] LustreError: 400270:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8227.913829] Lustre: server umount lustre-MDT0000 complete [ 8231.905975] LustreError: 399647:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8231.934083] LustreError: 399647:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 8232.555755] LustreError: 398903:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761291399 with bad export cookie 13396319341756377853 [ 8232.593588] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8232.800546] LustreError: 400471:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8232.845308] LustreError: 400471:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8233.410441] Lustre: server umount lustre-MDT0001 complete [ 8247.044883] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:36:51 (1761291411) [ 8255.526829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8256.643607] LustreError: 401193:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8256.748276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8261.805143] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8266.214650] LustreError: 401192:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8266.240230] LustreError: 401192:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 8273.165790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8273.492211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8277.743640] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8283.989064] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8288.333979] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8297.254921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8308.589255] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8317.561779] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8326.506943] Lustre: Permanently deactivating lustre-OST0000 [ 8326.543574] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8334.379226] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8336.669253] LustreError: 403838:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8336.676396] LustreError: 403838:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 8336.680063] LustreError: 403838:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8338.963661] LustreError: 403886:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 8338.976940] LustreError: 403886:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8347.346830] LustreError: 403983:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8347.349285] LustreError: 403983:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8347.598441] Lustre: server umount lustre-OST0000 complete [ 8351.137120] LustreError: 404136:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 8351.146037] LustreError: 404136:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8355.299468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8355.320293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8355.366235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8355.377980] Lustre: Skipped 2 previous similar messages [ 8364.541343] LustreError: 404233:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8364.574030] LustreError: 404233:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8366.638486] Lustre: server umount lustre-MDT0000 complete [ 8367.589679] LustreError: 401189:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8367.612122] LustreError: 401189:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 8371.049880] LustreError: 401173:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761291538 with bad export cookie 13396319341756378413 [ 8371.069458] LustreError: 401173:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8371.070034] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8381.608672] Lustre: server umount lustre-MDT0001 complete [ 8389.705182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8393.717417] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8396.119659] LustreError: 405280:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8402.428453] Lustre: 405621:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.133@tcp [ 8404.677488] LustreError: 405718:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8404.680383] LustreError: 405718:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8404.801680] Lustre: server umount MGS complete [ 8416.174105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8416.673674] LustreError: 406347:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8416.688163] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8416.690805] Lustre: Skipped 1 previous similar message [ 8416.738785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8416.741944] Lustre: Skipped 1 previous similar message [ 8420.473710] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8428.586030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8428.854335] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8432.182323] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8436.851254] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8439.817525] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8441.468224] Lustre: Permanently reactivating lustre-OST0000 [ 8441.484824] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8441.488339] Lustre: Skipped 2 previous similar messages [ 8471.521993] LustreError: 407976:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8477.741556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8481.671367] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8485.752871] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8496.614953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8496.630818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8496.635426] Lustre: Skipped 1 previous similar message [ 8496.640430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8496.643431] Lustre: Skipped 5 previous similar messages [ 8499.962361] LustreError: 409568:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8499.965391] LustreError: 409568:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 8500.126942] Lustre: server umount lustre-OST0000 complete [ 8505.830591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8505.834196] LustreError: Skipped 1 previous similar message [ 8509.541419] Lustre: server umount lustre-MDT0000 complete [ 8511.975053] LustreError: 407060:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8512.002479] LustreError: 407060:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 8512.552505] LustreError: 406326:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761291679 with bad export cookie 13396319341756379365 [ 8512.558810] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8512.569211] LustreError: 406326:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8513.047105] Lustre: server umount lustre-MDT0001 complete [ 8522.022684] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 8524.449628] Key type lgssc unregistered [ 8524.808181] LNet: 410506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8524.813655] LNetError: 410506:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8524.829933] LNet: Removed LNI 192.168.202.133@tcp [ 8525.661843] Key type .llcrypt unregistered [ 8525.669687] Key type ._llcrypt unregistered [ 8541.174862] Key type ._llcrypt registered [ 8541.177640] Key type .llcrypt registered [ 8541.305965] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 8563.982751] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 8565.699060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8565.762846] alg: No test for adler32 (adler32-zlib) [ 8566.884505] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 8567.201835] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 8568.896188] Key type lgssc registered [ 8570.115242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8579.066657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8584.811116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8590.340645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8596.726486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8603.151651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8603.209408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8603.229625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8604.593675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8604.644466] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8604.733504] Lustre: lustre-MDT0000: new disk, initializing [ 8604.845120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8604.888534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8608.427328] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8616.783401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8616.866745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8616.967242] Lustre: 413640:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8617.060919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8617.068455] Lustre: Skipped 1 previous similar message [ 8617.118527] Lustre: lustre-MDT0001: new disk, initializing [ 8617.200249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8617.253081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8617.260721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8617.360467] Lustre: 414436:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 8617.371320] Lustre: 414436:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 8617.383122] Lustre: 414436:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 8617.392092] Lustre: 414436:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 8617.403307] Lustre: 414436:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 8617.412151] Lustre: 414436:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 8620.827820] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8627.631756] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8630.810784] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8637.123209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8637.203338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8637.416065] Lustre: lustre-OST0000: new disk, initializing [ 8637.424772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8637.520394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8642.870021] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8643.687902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8643.719462] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8643.844481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8648.951724] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8651.997041] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8652.203632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8655.488781] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8655.662731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8659.428343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8659.430071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8659.431533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8659.438164] LustreError: Skipped 1 previous similar message [ 8659.452192] Lustre: Skipped 1 previous similar message [ 8663.375716] LustreError: 416581:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8663.607983] Lustre: server umount lustre-OST0000 complete [ 8668.645532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8668.649143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8668.658973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8668.661000] Lustre: Skipped 1 previous similar message [ 8669.668286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8669.670365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8669.700886] Lustre: Skipped 1 previous similar message [ 8673.181799] LustreError: 416782:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8673.185685] LustreError: 416782:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8673.267531] Lustre: server umount lustre-MDT0000 complete [ 8676.923766] LustreError: 413630:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761291844 with bad export cookie 1608125101386687868 [ 8676.929512] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8676.942523] LustreError: 413630:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8677.106499] LustreError: 416983:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8677.110046] LustreError: 416983:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8677.430467] Lustre: server umount lustre-MDT0001 complete [ 8687.085707] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8688.901964] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:44:14 (1761291854) [ 8696.295760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8696.634882] LustreError: 417705:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8696.664073] LustreError: 417705:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 8696.706975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8700.843566] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8701.921094] LustreError: 417705:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8706.033251] LustreError: 417705:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8708.373691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8708.740407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8712.427336] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8717.339808] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8720.884436] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8728.081591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8728.494640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8734.455255] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8739.978477] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8741.535639] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8743.802763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8749.027070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8749.030510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8749.039745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8751.079356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8753.243374] LustreError: 420143:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8753.246742] LustreError: 420143:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8753.484858] Lustre: server umount lustre-OST0000 complete [ 8759.780089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8759.783184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8759.814916] Lustre: Skipped 1 previous similar message [ 8759.825373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8759.829413] Lustre: Skipped 2 previous similar messages [ 8763.628258] LustreError: 420345:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8763.638472] LustreError: 420345:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8763.786321] Lustre: server umount lustre-MDT0000 complete [ 8766.439772] LustreError: 417700:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8766.468969] LustreError: 417700:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 8767.309609] LustreError: 417684:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761291934 with bad export cookie 1608125101386688792 [ 8767.313987] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8767.326640] LustreError: 417684:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8767.685319] Lustre: server umount lustre-MDT0001 complete [ 8776.649866] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 8778.623886] Key type lgssc unregistered [ 8778.831368] LNet: 421079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8778.838369] LNetError: 421079:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8779.884806] LNet: Removed LNI 192.168.202.133@tcp [ 8780.635975] Key type .llcrypt unregistered [ 8780.641833] Key type ._llcrypt unregistered [ 8801.229526] Key type ._llcrypt registered [ 8801.231669] Key type .llcrypt registered [ 8801.346638] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 8801.689334] Key type .llcrypt unregistered [ 8801.695486] Key type ._llcrypt unregistered [ 8817.351462] Key type ._llcrypt registered [ 8817.352830] Key type .llcrypt registered [ 8817.441708] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 8818.351984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8818.402237] alg: No test for adler32 (adler32-zlib) [ 8819.439871] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 8819.638371] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 8821.331561] Key type lgssc registered [ 8822.396571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8829.392740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8829.425704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8831.072401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8835.627250] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8841.187655] LustreError: 423254:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8841.195754] LustreError: 423254:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 8843.327906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8843.539088] LustreError: 423254:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8843.556632] LustreError: 423254:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 8843.560712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8848.241435] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8848.865854] LustreError: 423254:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8848.882248] LustreError: 423254:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 8853.749489] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8853.986896] LustreError: 423254:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8854.013240] LustreError: 423254:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 8856.058621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8861.068350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8861.156663] LustreError: 423253:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8861.182367] LustreError: 423253:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 8861.504150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8865.516625] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8869.817455] Lustre: 423250:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 8869.821478] Lustre: 423250:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 8869.830332] Lustre: 423250:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 8869.834855] Lustre: 423250:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 8869.851905] Lustre: 423250:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 8869.862849] Lustre: 423250:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 8875.652862] LustreError: 425309:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 8875.927143] Lustre: server umount lustre-OST0000 complete [ 8882.154646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8882.158135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8882.161799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8882.163872] Lustre: Skipped 1 previous similar message [ 8882.187305] Lustre: Skipped 1 previous similar message [ 8887.273048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8887.281979] Lustre: Skipped 1 previous similar message [ 8889.341673] LustreError: 425510:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8889.346770] LustreError: 425510:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8889.421581] Lustre: server umount lustre-MDT0000 complete [ 8892.388916] LustreError: 423254:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8892.879814] LustreError: 424873:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761292060 with bad export cookie 619147371821587907 [ 8892.883629] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8892.892459] LustreError: 424873:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8903.148111] LustreError: 425712:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8903.167775] LustreError: 425712:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8903.433449] Lustre: server umount lustre-MDT0001 complete [ 8912.994171] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 8915.576556] Key type lgssc unregistered [ 8915.849816] LNet: 426247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8915.859566] LNetError: 426247:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8916.903656] LNet: Removed LNI 192.168.202.133@tcp [ 8917.757982] Key type .llcrypt unregistered [ 8917.759834] Key type ._llcrypt unregistered [ 8940.765592] Key type ._llcrypt registered [ 8940.767276] Key type .llcrypt registered [ 8940.874114] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 8942.019982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8942.055864] alg: No test for adler32 (adler32-zlib) [ 8943.134440] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 8943.334739] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 8945.024208] Key type lgssc registered [ 8946.151840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8952.823729] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8952.839995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8968.352453] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8973.472224] LustreError: 427622:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 9000.163191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9004.204559] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9008.093001] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9013.070457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9013.456112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9013.518644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9016.690532] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9020.901772] LustreError: 428545:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9020.914030] LustreError: 428545:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 9023.209493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9023.552104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9026.975643] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9031.795245] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9035.062605] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9038.066031] Lustre: 429814:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 9038.074784] Lustre: 429814:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 9038.083425] Lustre: 429814:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 9038.089311] Lustre: 429814:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 9038.093977] Lustre: 429814:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 9038.097083] Lustre: 429814:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 9043.262388] LustreError: 429905:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 9043.427655] Lustre: server umount lustre-OST0000 complete [ 9049.059134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9049.061776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9049.063152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9049.085123] Lustre: Skipped 1 previous similar message [ 9054.177138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9054.190748] Lustre: Skipped 2 previous similar messages [ 9056.235204] LustreError: 430106:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 9056.257736] LustreError: 430106:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9056.330067] Lustre: server umount lustre-MDT0000 complete [ 9059.300792] LustreError: 428551:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9059.320812] LustreError: 428551:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 9059.530516] LustreError: 427629:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761292226 with bad export cookie 1577143759770965249 [ 9059.534511] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9059.558112] LustreError: 427629:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9069.115764] LustreError: 430309:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9069.121314] LustreError: 430309:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9069.413646] Lustre: server umount lustre-MDT0001 complete [ 9079.775929] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 9082.667577] Key type lgssc unregistered [ 9082.974887] LNet: 430842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9082.984630] LNetError: 430842:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9083.005578] LNet: Removed LNI 192.168.202.133@tcp [ 9084.128898] Key type .llcrypt unregistered [ 9084.130927] Key type ._llcrypt unregistered [ 9106.099553] Key type ._llcrypt registered [ 9106.101350] Key type .llcrypt registered [ 9106.214256] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 9107.070732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9107.078367] alg: No test for adler32 (adler32-zlib) [ 9108.140718] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 9108.343695] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 9109.988586] Key type lgssc registered [ 9111.063957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9117.565244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9117.590888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9118.947230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9122.071679] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9129.388158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9129.442109] LustreError: 432235:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9129.447600] LustreError: 432235:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 9129.786544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9133.302681] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9138.684383] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9142.230961] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9149.486323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9149.787283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9154.612891] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9159.196688] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9164.149264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9169.382528] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9169.386380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9169.401978] Lustre: Skipped 2 previous similar messages [ 9169.418902] Lustre: Skipped 2 previous similar messages [ 9170.932773] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 9173.483680] LustreError: 434484:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9173.559655] Lustre: server umount lustre-MDT0000 complete [ 9174.498360] LustreError: 432232:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9174.519133] LustreError: 432232:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 9175.377767] LustreError: 434485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.33@tcp arrived at 1761292342 with bad export cookie 17057234041579536601 [ 9175.396390] LustreError: 434485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9176.919713] LustreError: 434485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761292344 with bad export cookie 17057234041579535936 [ 9176.925555] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9176.928803] LustreError: 434485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9179.618145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9179.622389] LustreError: 432963:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9179.623888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9179.627341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9179.659866] LustreError: 432963:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 9181.666326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9181.666436] LustreError: 432963:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9181.687145] LustreError: 432963:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 9183.048979] LustreError: 434687:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9183.065155] LustreError: 434687:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9183.343466] Lustre: server umount lustre-OST0000 complete [ 9186.786060] LustreError: 432963:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9186.795769] LustreError: 432963:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 9190.173253] LustreError: 435041:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9190.187221] LustreError: 435041:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9190.413572] Lustre: server umount lustre-MDT0001 complete [ 9199.590759] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 9202.820238] Key type lgssc unregistered [ 9203.120627] LNet: 435574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9203.136778] LNetError: 435574:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9204.211496] LNet: Removed LNI 192.168.202.133@tcp [ 9204.994058] Key type .llcrypt unregistered [ 9204.995759] Key type ._llcrypt unregistered [ 9227.309891] Key type ._llcrypt registered [ 9227.311855] Key type .llcrypt registered [ 9227.407082] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 9228.030114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9228.050460] alg: No test for adler32 (adler32-zlib) [ 9229.097818] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 9229.367537] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 9231.016174] Key type lgssc registered [ 9232.126587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9239.189273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9239.211121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9240.587051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9244.383720] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9250.787560] LustreError: 436972:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9250.811359] LustreError: 436972:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 9252.421458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9252.944447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9256.858235] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9261.014266] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9263.556886] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9268.806086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9269.053360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9273.887479] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9279.081496] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9285.439729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9293.794152] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9293.805640] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9293.817678] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9295.345133] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 9295.852076] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9298.604504] LustreError: 439242:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9298.675713] Lustre: server umount lustre-MDT0001 complete [ 9300.464569] LustreError: 436966:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9306.081801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9306.088429] LustreError: 436972:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9306.092179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9306.108261] LustreError: 436972:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 9306.124692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9306.140958] Lustre: Skipped 2 previous similar messages [ 9310.300230] LustreError: 439448:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9310.304553] LustreError: 439448:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9310.524422] Lustre: server umount lustre-OST0000 complete [ 9311.202406] LustreError: 436972:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9314.525847] LustreError: 439649:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9314.546194] LustreError: 439649:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9314.902789] Lustre: server umount lustre-MDT0000 complete [ 9327.561531] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 9330.102813] Key type lgssc unregistered [ 9330.387838] LNet: 440333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9330.397177] LNetError: 440333:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9331.441272] LNet: Removed LNI 192.168.202.133@tcp [ 9332.509836] Key type .llcrypt unregistered [ 9332.513605] Key type ._llcrypt unregistered [ 9356.613703] Key type ._llcrypt registered [ 9356.615427] Key type .llcrypt registered [ 9356.747720] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 9369.898909] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 9370.970965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9370.998160] alg: No test for adler32 (adler32-zlib) [ 9372.099526] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 9372.277900] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 9373.920758] Key type lgssc registered [ 9375.018707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9382.919275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9388.373532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9393.986655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9400.014241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9409.480925] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 9419.551051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9427.540551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9427.646824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9427.692986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9429.009921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9429.062756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9429.199480] Lustre: lustre-MDT0000: new disk, initializing [ 9429.338774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9429.357231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9434.131230] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9442.936770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9450.140109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9450.212520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9450.435698] Lustre: lustre-OST0000: new disk, initializing [ 9450.438972] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9450.441551] Lustre: Skipped 1 previous similar message [ 9450.524996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9456.067283] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9460.341493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9460.347908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9460.376440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9462.859613] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9472.105823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9478.347746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9478.413889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9478.514801] Lustre: 444310:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9478.580362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9478.648874] Lustre: lustre-MDT0001: new disk, initializing [ 9478.805554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9478.870329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 9478.887889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 9478.958629] Lustre: 446550:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 9478.978478] Lustre: 446550:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 9478.984917] Lustre: 446550:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 9479.000852] Lustre: 446550:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 9479.019969] Lustre: 446550:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 9479.035782] Lustre: 446550:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 9482.483278] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9490.786698] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9491.057148] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9495.349428] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9495.619507] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9500.665781] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 9502.178680] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9502.187646] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9502.206375] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9502.222932] Lustre: Skipped 1 previous similar message [ 9502.234660] Lustre: Skipped 1 previous similar message [ 9505.759591] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 9506.373145] LustreError: 447732:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9506.529289] Lustre: server umount lustre-MDT0001 complete [ 9512.421792] LustreError: 445918:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9512.458066] LustreError: 445918:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 9517.544987] LustreError: 444317:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9517.566922] LustreError: 444317:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 9522.663711] LustreError: 444316:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9522.688048] LustreError: 444316:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 9526.757592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9526.761513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9526.783453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9527.778764] LustreError: 444321:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9529.548184] LustreError: 448003:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 9529.559743] LustreError: 448003:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9529.863236] Lustre: server umount lustre-OST0000 complete [ 9532.912365] LustreError: 444321:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9539.421026] LustreError: 448205:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 9539.424114] LustreError: 448205:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9539.796888] Lustre: server umount lustre-MDT0000 complete [ 9552.838292] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 9555.487414] Key type lgssc unregistered [ 9555.819401] LNet: 448893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9555.833144] LNetError: 448893:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9555.847026] LNet: Removed LNI 192.168.202.133@tcp [ 9556.586828] Key type .llcrypt unregistered [ 9556.588654] Key type ._llcrypt unregistered [ 9595.980114] Key type ._llcrypt registered [ 9595.981708] Key type .llcrypt registered [ 9596.106706] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 9596.435383] Key type .llcrypt unregistered [ 9596.444297] Key type ._llcrypt unregistered [ 9611.415427] Key type ._llcrypt registered [ 9611.416894] Key type .llcrypt registered [ 9611.517193] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 9612.438537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9612.550945] alg: No test for adler32 (adler32-zlib) [ 9613.658604] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [ 9613.837765] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 9615.449164] Key type lgssc registered [ 9616.506280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9623.278437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9628.477579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9633.974693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9639.790670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9639.843676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9639.855517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9641.251255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9641.307674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9641.381342] Lustre: lustre-MDT0000: new disk, initializing [ 9641.516115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9641.542481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9645.583168] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9655.441920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9655.495927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9655.593585] Lustre: 452241:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9655.652679] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9655.656295] Lustre: Skipped 1 previous similar message [ 9655.718251] Lustre: lustre-MDT0001: new disk, initializing [ 9655.818929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9655.857172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9655.866387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9655.922704] Lustre: 453039:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 9655.932204] Lustre: 453039:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 9655.939875] Lustre: 453039:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 9655.942905] Lustre: 453039:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 9655.947563] Lustre: 453039:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 9655.952615] Lustre: 453039:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 9659.435454] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9667.842738] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9671.300634] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9677.469946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9677.525507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9677.765721] Lustre: lustre-OST0000: new disk, initializing [ 9677.768537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9677.847596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9682.993726] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9683.040084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9683.048444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9683.211303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9690.364213] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9696.739718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9696.747578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9696.758894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9698.788606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9698.791138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9702.301854] LustreError: 454694:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9702.434899] Lustre: server umount lustre-MDT0000 complete [ 9705.757525] LustreError: 452233:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761292872 with bad export cookie 13705847282021916292 [ 9705.759412] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9705.769995] LustreError: 452233:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9705.935065] LustreError: 454896:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9705.949210] LustreError: 454896:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9706.202255] Lustre: server umount lustre-MDT0001 complete [ 9719.621565] LustreError: 455097:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9719.628289] LustreError: 455097:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9719.814698] Lustre: server umount lustre-OST0000 complete [ 9728.959578] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:01:34 (1761292894) [ 9730.054375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9736.237132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9736.559190] LustreError: 455926:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9736.627924] LustreError: 455926:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 4 previous similar messages [ 9736.683340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9739.599955] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9741.804341] LustreError: 455926:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9745.895676] LustreError: 455926:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9747.381142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9747.675335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9750.707797] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9754.643665] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9757.468560] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9763.090227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9763.162882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9763.302985] Lustre: Found index 0 for lustre-OST0000, updating log [ 9763.473099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9767.434534] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9771.992840] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9780.845757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 9786.339197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9786.354754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9786.371710] Lustre: Skipped 1 previous similar message [ 9786.377635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9786.380755] Lustre: Skipped 2 previous similar messages [ 9788.222121] LustreError: 458229:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9788.233613] LustreError: 458229:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9788.386266] Lustre: server umount lustre-OST0000 complete [ 9790.850848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9792.481511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9792.488510] Lustre: Skipped 1 previous similar message [ 9792.493959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9792.501600] Lustre: Skipped 4 previous similar messages [ 9798.557178] LustreError: 458484:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9798.560092] LustreError: 458484:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9798.626507] LustreError: 455921:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9798.635941] LustreError: 455921:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 9798.655712] Lustre: server umount lustre-MDT0000 complete [ 9801.765615] LustreError: 455908:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761292968 with bad export cookie 13705847282021917727 [ 9801.779257] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9801.790924] LustreError: 455908:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9802.177675] Lustre: server umount lustre-MDT0001 complete [ 9809.852740] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:02:55 (1761292975) [ 9810.768527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9811.689821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9812.500878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9813.372612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9814.381298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9815.353401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9816.372183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9817.425214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9818.366519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9819.487455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9820.696356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9822.054887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9823.509947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9824.725804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9825.678926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9826.909339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9827.974256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9829.085762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9829.972137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9830.914889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9831.748608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9832.691387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9833.717350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9834.722172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9835.563852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9836.402899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9837.146456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9837.984667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9838.700927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9839.495658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9840.138968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9840.773714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9841.435866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9842.155849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9842.885766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9843.561464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9844.297929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9844.982938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9845.697651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9846.387841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9847.010294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9847.669572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9848.342774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9848.974567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9849.733791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9850.455462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9851.177540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9851.906946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9852.637291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9853.409031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9854.077516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9860.123538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9860.487230] LustreError: 462083:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9860.579911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9863.528457] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9869.433858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9873.064598] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9877.992627] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9881.613104] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9887.461513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9887.550389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9887.748681] Lustre: Found index 0 for lustre-OST0000, updating log [ 9892.499458] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9897.300232] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9901.537362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9901.540707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9901.555406] Lustre: Skipped 1 previous similar message [ 9901.566243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9901.568787] Lustre: Skipped 2 previous similar messages [ 9905.026071] LustreError: 464350:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9905.043953] LustreError: 464350:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9905.252546] Lustre: server umount lustre-OST0000 complete [ 9910.374631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9910.377266] Lustre: Skipped 2 previous similar messages [ 9914.640801] Lustre: server umount lustre-MDT0000 complete [ 9915.503959] LustreError: 462079:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9915.522283] LustreError: 462079:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 9917.526859] LustreError: 462062:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761293084 with bad export cookie 13705847282021918805 [ 9917.531907] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9917.535044] LustreError: 462062:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9917.832790] Lustre: server umount lustre-MDT0001 complete [ 9920.135269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9933.683512] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 9942.207801] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 9952.334657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9959.299779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9965.769215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9973.159866] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9980.613315] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 04:05:46 (1761293146) [ 9997.794691] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [10000.223938] Key type lgssc unregistered [10000.519399] LNet: 468321:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10000.530195] LNetError: 468321:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10032.296450] LNet: Removed LNI 192.168.202.133@tcp [10032.927228] Key type .llcrypt unregistered [10032.928945] Key type ._llcrypt unregistered [10045.442597] Key type ._llcrypt registered [10045.444395] Key type .llcrypt registered [10045.547159] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [10046.244112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10046.367632] alg: No test for adler32 (adler32-zlib) [10047.434942] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [10047.673664] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [10049.353035] Key type lgssc registered [10050.334712] Lustre: Echo OBD driver; http://www.lustre.org/ [10053.980385] LNet: Added LNI 192.168.202.133@tcp100 [8/256/0/180] [10055.497256] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [10072.033925] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [10074.744583] Key type lgssc unregistered [10075.121514] LNet: 470171:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10075.127948] LNetError: 470171:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10075.134212] LNet: Removed LNI 192.168.202.133@tcp [10075.145174] LNet: Removed LNI 192.168.202.133@tcp100 [10075.843234] Key type .llcrypt unregistered [10075.844715] Key type ._llcrypt unregistered [10096.527227] Key type ._llcrypt registered [10096.528790] Key type .llcrypt registered [10096.643958] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [10110.755801] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [10111.625277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10111.752380] alg: No test for adler32 (adler32-zlib) [10112.763317] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [10112.942383] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [10114.568215] Key type lgssc registered [10115.811910] Lustre: Echo OBD driver; http://www.lustre.org/ [10123.957333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10130.281529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10136.758826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10144.170080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10150.969651] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:08:36 (1761293316) [10166.229095] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [10168.352916] Key type lgssc unregistered [10168.663568] LNet: 474216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10168.668239] LNetError: 474216:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10168.679885] LNet: Removed LNI 192.168.202.133@tcp [10169.272684] Key type .llcrypt unregistered [10169.276840] Key type ._llcrypt unregistered [10181.444161] Key type ._llcrypt registered [10181.445805] Key type .llcrypt registered [10181.572264] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [10182.391501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10182.422325] alg: No test for adler32 (adler32-zlib) [10183.419268] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [10183.593633] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [10185.240232] Key type lgssc registered [10186.251524] Lustre: Echo OBD driver; http://www.lustre.org/ [10194.421562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10194.461887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10194.472949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10195.640486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10195.666317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10195.718695] Lustre: lustre-MDT0000: new disk, initializing [10195.781017] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10195.801887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10198.552207] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10206.872148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10206.933807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10207.023697] Lustre: 475388:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10207.082518] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10207.085374] Lustre: Skipped 1 previous similar message [10207.137019] Lustre: lustre-MDT0001: new disk, initializing [10207.215412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10207.255478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10207.269285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10207.328797] Lustre: 476185:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [10207.339268] Lustre: 476185:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [10207.353972] Lustre: 476185:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [10207.369727] Lustre: 476185:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [10207.377477] Lustre: 476185:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [10207.382918] Lustre: 476185:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [10210.597974] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10216.850254] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10219.829542] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10224.791496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10224.838155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10225.010436] Lustre: lustre-OST0000: new disk, initializing [10225.013958] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10225.092864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10229.740693] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10230.819327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10230.835852] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10230.946311] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10236.014194] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10238.391854] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing lnet_if_list [10241.302975] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [10241.326784] LNet: Added LNI 192.168.202.133@tcp42 [8/256/0/180] [10256.872374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10256.872829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10256.878754] LustreError: Skipped 1 previous similar message [10256.879544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10256.892577] Lustre: Skipped 1 previous similar message [10259.520890] LustreError: 478773:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10259.644752] Lustre: server umount lustre-OST0000 complete [10263.521336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10263.524842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10263.535743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10263.538718] Lustre: Skipped 1 previous similar message [10267.107937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10267.109117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10267.134585] Lustre: Skipped 1 previous similar message [10268.632779] LustreError: 478974:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10268.635678] LustreError: 478974:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10268.692181] Lustre: server umount lustre-MDT0000 complete [10271.048453] LustreError: 475381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761293438 with bad export cookie 8139612335809369382 [10271.053210] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10271.057158] LustreError: 475381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10271.137653] LustreError: 479177:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10271.140580] LustreError: 479177:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10271.310358] Lustre: server umount lustre-MDT0001 complete [10278.097800] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [10279.830674] Key type lgssc unregistered [10280.059963] LNet: 479715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10280.064676] LNetError: 479715:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10280.069028] LNet: Removed LNI 192.168.202.133@tcp [10280.074567] LNet: Removed LNI 192.168.202.133@tcp42 [10280.525635] Key type .llcrypt unregistered [10280.528393] Key type ._llcrypt unregistered [10292.567677] Key type ._llcrypt registered [10292.570028] Key type .llcrypt registered [10292.625499] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [10293.152058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10293.195794] alg: No test for adler32 (adler32-zlib) [10294.120268] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [10294.261853] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [10295.888227] Key type lgssc registered [10296.510676] Lustre: Echo OBD driver; http://www.lustre.org/ [10301.619675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10301.632160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10302.944494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10305.364702] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10309.942910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10310.165870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10312.202268] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10315.140942] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10317.678255] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10322.533652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10322.850072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10326.419682] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10329.941994] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10342.115803] loop0: detected capacity change from 0 to 204800000 [10342.231464] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10342.236693] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10342.246588] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10342.278081] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10345.704532] loop0: detected capacity change from 0 to 204800000 [10345.797115] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10345.802520] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10345.808235] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10345.887762] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10349.050482] loop0: detected capacity change from 0 to 204800000 [10352.459267] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10352.499685] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10352.571736] Lustre: fs15246-OST0000: new disk, initializing [10352.575137] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [10352.611959] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10356.553870] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10363.973385] loop1: detected capacity change from 0 to 204800000 [10368.496660] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10368.551409] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10368.613049] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [10368.657880] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [10368.714562] Lustre: fs15246-MDT0000: new disk, initializing [10368.834844] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10368.851754] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10368.939580] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10369.957132] Lustre: 485312:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@0000000056cdefe1 x1846850052201600/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1761293553 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [10371.419435] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10375.010709] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10375.024049] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10375.047921] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10384.871110] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10384.883754] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10389.989270] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10390.607094] LustreError: 486437:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [10390.664782] Lustre: server umount fs15246-MDT0000 complete [10395.111435] LustreError: 480929:0:(ldlm_lib.c:1245:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10395.121430] LustreError: 480929:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [10396.857355] LustreError: 486938:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [10396.864966] LustreError: 486938:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10396.889203] Lustre: server umount fs15246-OST0000 complete [10404.834696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10404.839133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10404.847376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10409.731135] LustreError: 487441:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10409.735747] LustreError: 487441:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10409.852496] Lustre: server umount lustre-OST0000 complete [10412.513500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10412.517452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10412.526041] Lustre: Skipped 1 previous similar message [10412.529586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10412.532554] Lustre: Skipped 2 previous similar messages [10415.587375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10420.707667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10420.710907] Lustre: Skipped 3 previous similar messages [10426.336242] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10426.395816] LustreError: 487644:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10426.399422] LustreError: 487644:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10426.436146] Lustre: server umount lustre-MDT0000 complete [10428.591839] LustreError: 480916:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761293595 with bad export cookie 12130349313763748125 [10428.597267] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10428.620085] LustreError: 480916:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10428.849105] Lustre: server umount lustre-MDT0001 complete [10435.425250] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [10437.217255] Key type lgssc unregistered [10437.398228] LNet: 488380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10437.407989] LNetError: 488380:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10437.425291] LNet: Removed LNI 192.168.202.133@tcp [10437.995652] Key type .llcrypt unregistered [10437.998306] Key type ._llcrypt unregistered [10449.669368] Key type ._llcrypt registered [10449.671594] Key type .llcrypt registered [10449.748849] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [10450.472980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10450.487742] alg: No test for adler32 (adler32-zlib) [10451.463229] Lustre: Lustre: Build Version: 2.16.59_47_g5ba01f8 [10451.605961] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [10453.224179] Key type lgssc registered [10453.986967] Lustre: Echo OBD driver; http://www.lustre.org/ [10458.283017] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10461.063950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10463.923614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10466.598746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10477.225799] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [10482.906569] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [10488.299266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10491.725921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10495.316268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10499.487475] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10503.961084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10503.996125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10504.009686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10505.161737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10505.198549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10505.281105] Lustre: lustre-MDT0000: new disk, initializing [10505.335341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10505.348724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10507.486667] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10514.440165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10514.496496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10514.557635] Lustre: 492788:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10514.610348] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10514.613471] Lustre: Skipped 1 previous similar message [10514.684952] Lustre: lustre-MDT0001: new disk, initializing [10514.769269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10514.799165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10514.809317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10514.895975] Lustre: 493587:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [10514.910247] Lustre: 493587:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [10514.916480] Lustre: 493587:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [10514.926661] Lustre: 493587:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [10514.934766] Lustre: 493587:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [10514.941725] Lustre: 493587:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [10517.423371] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10521.878930] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10523.878193] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10527.425628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10527.463189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10527.595392] Lustre: lustre-OST0000: new disk, initializing [10527.600675] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10527.651996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10531.074884] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10531.419309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10531.425223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10531.491657] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10535.812339] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10538.175452] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10538.330430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10540.397499] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10540.514786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10542.051807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10542.053598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10542.059160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10542.064658] LustreError: Skipped 1 previous similar message [10542.080761] Lustre: Skipped 1 previous similar message [10547.171837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10547.175935] Lustre: Skipped 1 previous similar message [10552.292256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10552.295152] Lustre: Skipped 1 previous similar message [10555.872222] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10555.912578] LustreError: 495727:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10556.055367] Lustre: server umount lustre-OST0000 complete [10560.993505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10560.997235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10561.003318] Lustre: Skipped 1 previous similar message [10561.007212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10562.535311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10564.749843] LustreError: 495930:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10564.752272] LustreError: 495930:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10564.793467] Lustre: server umount lustre-MDT0000 complete [10567.081560] LustreError: 492779:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761293734 with bad export cookie 2976371565584746463 [10567.082836] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10567.086520] LustreError: 492779:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10567.202202] LustreError: 496131:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10567.205944] LustreError: 496131:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10567.322887] Lustre: server umount lustre-MDT0001 complete [10579.192712] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [10583.846021] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [10588.937072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10592.254122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10595.190929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10598.175395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10598.939391] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10450 sec ========= 04:16:05 (1761293765) [10599.691566] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:16:05 (1761293765) === [10601.159687] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:16:07 (1761293767) === [10619.012324] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [10620.271769] Key type lgssc unregistered [10620.425726] LNet: 500981:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10620.429417] LNetError: 500981:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10620.437883] LNet: Removed LNI 192.168.202.133@tcp [10620.848416] Key type .llcrypt unregistered [10620.850339] Key type ._llcrypt unregistered