[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 484760905 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003285] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007772] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.012122] LSM: Security Framework initializing [ 0.013073] Yama: becoming mindful. [ 0.014049] SELinux: Initializing. [ 0.016048] *** VALIDATE selinux *** [ 0.025798] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030263] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031166] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033042] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034101] *** VALIDATE tmpfs *** [ 0.035509] *** VALIDATE proc *** [ 0.036312] *** VALIDATE cgroup *** [ 0.037011] *** VALIDATE cgroup2 *** [ 0.038269] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040105] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042030] Spectre V2 : User space: Vulnerable [ 0.043011] Speculative Store Bypass: Vulnerable [ 0.046029] debug: unmapping init [mem 0xffffffff9aa59000-0xffffffff9aa60fff] [ 0.048473] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050330] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051022] ... version: 2 [ 0.052013] ... bit width: 48 [ 0.053014] ... generic registers: 4 [ 0.054012] ... value mask: 0000ffffffffffff [ 0.055014] ... max period: 00007fffffffffff [ 0.056013] ... fixed-purpose events: 3 [ 0.057013] ... event mask: 000000070000000f [ 0.059235] rcu: Hierarchical SRCU implementation. [ 0.061406] smp: Bringing up secondary CPUs ... [ 0.062560] x86: Booting SMP configuration: [ 0.063029] .... node #0, CPUs: #1 #2 #3 [ 0.066590] smp: Brought up 1 node, 4 CPUs [ 0.068012] smpboot: Max logical packages: 1 [ 0.069018] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.277145] node 0 deferred pages initialised in 205ms [ 0.280300] devtmpfs: initialized [ 0.281226] x86/mm: Memory block size: 128MB [ 0.284091] gcov: version magic: 0x41383552 [ 0.285630] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.289086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.290000] pinctrl core: initialized pinctrl subsystem [ 0.291193] [ 0.291741] ************************************************************* [ 0.294014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.296012] ** ** [ 0.298015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.300012] ** ** [ 0.302022] ** This means that this kernel is built to expose internal ** [ 0.305014] ** IOMMU data structures, which may compromise security on ** [ 0.307010] ** your system. ** [ 0.308006] ** ** [ 0.310011] ** If you see this message and you are not debugging the ** [ 0.312012] ** kernel, report this immediately to your vendor! ** [ 0.314013] ** ** [ 0.316011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.318010] ************************************************************* [ 0.320613] NET: Registered protocol family 16 [ 0.322402] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.325079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.327061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.330043] cpuidle: using governor menu [ 0.331852] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.334628] PCI: Using configuration type 1 for base access [ 0.336139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.345153] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.348108] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.351067] cryptd: max_cpu_qlen set to 1000 [ 0.353263] ACPI: Added _OSI(Module Device) [ 0.355019] ACPI: Added _OSI(Processor Device) [ 0.356000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.357018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.362125] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.367511] ACPI: Interpreter enabled [ 0.369056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.370008] ACPI: Using IOAPIC for interrupt routing [ 0.371106] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.374375] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.384057] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.386040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.389022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.392083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.396245] acpiphp: Slot [2] registered [ 0.398126] acpiphp: Slot [3] registered [ 0.399093] acpiphp: Slot [4] registered [ 0.401123] acpiphp: Slot [5] registered [ 0.403141] acpiphp: Slot [6] registered [ 0.404128] acpiphp: Slot [7] registered [ 0.405105] acpiphp: Slot [8] registered [ 0.406087] acpiphp: Slot [9] registered [ 0.408090] acpiphp: Slot [10] registered [ 0.409094] acpiphp: Slot [11] registered [ 0.410093] acpiphp: Slot [12] registered [ 0.412081] acpiphp: Slot [13] registered [ 0.413105] acpiphp: Slot [14] registered [ 0.414099] acpiphp: Slot [15] registered [ 0.416106] acpiphp: Slot [16] registered [ 0.417100] acpiphp: Slot [17] registered [ 0.419113] acpiphp: Slot [18] registered [ 0.420093] acpiphp: Slot [19] registered [ 0.421074] acpiphp: Slot [20] registered [ 0.422078] acpiphp: Slot [21] registered [ 0.423100] acpiphp: Slot [22] registered [ 0.425090] acpiphp: Slot [23] registered [ 0.426092] acpiphp: Slot [24] registered [ 0.427087] acpiphp: Slot [25] registered [ 0.429080] acpiphp: Slot [26] registered [ 0.430105] acpiphp: Slot [27] registered [ 0.432118] acpiphp: Slot [28] registered [ 0.433086] acpiphp: Slot [29] registered [ 0.434100] acpiphp: Slot [30] registered [ 0.436106] acpiphp: Slot [31] registered [ 0.437084] PCI host bridge to bus 0000:00 [ 0.439016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.440016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.443024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.445025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.447024] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.450027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.452153] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.456396] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.460000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.469018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.475707] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.479022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.482017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.484016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.486499] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.490086] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.492047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.495491] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.501015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.513017] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.518000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.522374] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.527013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.533017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.551021] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.561619] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.570017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.574017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.592015] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.601684] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.615018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.622020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.639021] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.648756] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.654016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.666019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.686019] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.702787] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.710016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.715019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.733026] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.742483] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.747014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.752013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.765085] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.775260] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.778441] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.780377] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.782379] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.785241] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.789128] iommu: Default domain type: Passthrough [ 0.791409] SCSI subsystem initialized [ 0.792148] ACPI: bus type USB registered [ 0.794088] usbcore: registered new interface driver usbfs [ 0.795077] usbcore: registered new interface driver hub [ 0.797099] usbcore: registered new device driver usb [ 0.798127] pps_core: LinuxPPS API ver. 1 registered [ 0.800009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.802051] PTP clock support registered [ 0.804134] EDAC MC: Ver: 3.0.0 [ 0.805732] PCI: Using ACPI for IRQ routing [ 0.808340] NetLabel: Initializing [ 0.809011] NetLabel: domain hash size = 128 [ 0.811013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.813080] NetLabel: unlabeled traffic allowed by default [ 0.816040] vgaarb: loaded [ 0.818116] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.819012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.822607] clocksource: Switched to clocksource kvm-clock [ 0.930403] VFS: Disk quotas dquot_6.6.0 [ 0.932270] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.934910] *** VALIDATE ramfs *** [ 0.936133] *** VALIDATE hugetlbfs *** [ 0.937593] pnp: PnP ACPI init [ 0.940619] pnp: PnP ACPI: found 6 devices [ 0.960464] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.963719] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.966066] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.968472] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.971081] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.973677] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.976788] NET: Registered protocol family 2 [ 0.979293] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.983739] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.986978] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.991343] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.994548] TCP: Hash tables configured (established 65536 bind 65536) [ 0.997474] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.000341] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.003212] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.005889] NET: Registered protocol family 1 [ 1.008449] RPC: Registered named UNIX socket transport module. [ 1.010373] RPC: Registered udp transport module. [ 1.012457] RPC: Registered tcp transport module. [ 1.014229] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.016414] NET: Registered protocol family 44 [ 1.017791] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.023228] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.025443] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.027802] PCI: CLS 0 bytes, default 64 [ 1.030083] Unpacking initramfs... [ 2.424854] debug: unmapping init [mem 0xffff8caabcc54000-0xffff8caabffbffff] [ 2.428812] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.430965] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.433880] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.938238] Initialise system trusted keyrings [ 2.939869] Key type blacklist registered [ 2.941988] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.951272] zbud: loaded [ 2.954216] *** VALIDATE nfs *** [ 2.955410] *** VALIDATE nfs4 *** [ 2.959896] pstore: using deflate compression [ 2.963485] Platform Keyring initialized [ 3.076689] NET: Registered protocol family 38 [ 3.078532] Key type asymmetric registered [ 3.080649] Asymmetric key parser 'x509' registered [ 3.082685] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.085132] io scheduler mq-deadline registered [ 3.086762] io scheduler kyber registered [ 3.088197] io scheduler bfq registered [ 3.090198] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.094374] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.096846] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.100064] ACPI: Power Button [PWRF] [ 3.197860] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.296877] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.492647] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.590167] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.776962] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.807376] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.840194] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.849969] Non-volatile memory driver v1.3 [ 3.851814] Linux agpgart interface v0.103 [ 3.897334] virtio_blk virtio1: [vda] 132696 512-byte logical blocks (67.9 MB/64.8 MiB) [ 3.901664] vda: detected capacity change from 0 to 67940352 [ 3.915770] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.919118] vdb: detected capacity change from 0 to 1073741824 [ 3.935651] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.939155] vdc: detected capacity change from 0 to 2621440000 [ 3.952417] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.955200] vdd: detected capacity change from 0 to 2621440000 [ 3.969454] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.972475] vde: detected capacity change from 0 to 4294967296 [ 3.987421] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.989769] vdf: detected capacity change from 0 to 4294967296 [ 3.996128] libphy: Fixed MDIO Bus: probed [ 4.004679] usbcore: registered new interface driver usbserial_generic [ 4.007231] usbserial: USB Serial support registered for generic [ 4.009496] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.013564] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.015584] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.018240] mousedev: PS/2 mouse device common for all mice [ 4.021611] rtc_cmos 00:05: RTC can wake from S4 [ 4.023858] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.025589] rtc_cmos 00:05: registered as rtc0 [ 4.028903] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.031770] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.033047] intel_pstate: CPU model not supported [ 4.037116] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.041125] hid: raw HID events driver (C) Jiri Kosina [ 4.043469] usbcore: registered new interface driver usbhid [ 4.045563] usbhid: USB HID core driver [ 4.047145] drop_monitor: Initializing network drop monitor service [ 4.049691] Initializing XFRM netlink socket [ 4.051735] NET: Registered protocol family 10 [ 4.055923] Segment Routing with IPv6 [ 4.057202] NET: Registered protocol family 17 [ 4.058954] mpls_gso: MPLS GSO support [ 4.065650] RAS: Correctable Errors collector initialized. [ 4.068276] AVX version of gcm_enc/dec engaged. [ 4.069895] AES CTR mode by8 optimization enabled [ 4.154599] sched_clock: Marking stable (4154559992, 0)->(5052316310, -897756318) [ 4.159472] registered taskstats version 1 [ 4.161447] Loading compiled-in X.509 certificates [ 4.163278] zswap: loaded using pool lzo/zbud [ 4.195194] Key type big_key registered [ 4.206989] Key type encrypted registered [ 4.208741] ima: No TPM chip found, activating TPM-bypass! [ 4.210877] ima: Allocated hash algorithm: sha1 [ 4.212276] ima: No architecture policies found [ 4.213475] evm: Initialising EVM extended attributes: [ 4.215221] evm: security.selinux [ 4.216579] evm: security.ima [ 4.217777] evm: security.capability [ 4.219254] evm: HMAC attrs: 0x1 [ 4.221945] rtc_cmos 00:05: setting system clock to 2025-07-28 22:23:39 UTC (1753741419) [ 4.227872] debug: unmapping init [mem 0xffffffff9ba03000-0xffffffff9bbfffff] [ 4.230662] debug: unmapping init [mem 0xffffffff9a782000-0xffffffff9aa58fff] [ 4.238133] Write protecting the kernel read-only data: 28672k [ 4.242159] debug: unmapping init [mem 0xffffffff98e03000-0xffffffff98ffffff] [ 4.245262] debug: unmapping init [mem 0xffffffff99714000-0xffffffff997fffff] [ 4.285717] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.293409] systemd[1]: Detected virtualization kvm. [ 4.295237] systemd[1]: Detected architecture x86-64. [ 4.297445] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.323834] systemd[1]: No hostname configured. [ 4.325542] systemd[1]: Set hostname to . [ 4.327580] random: systemd: uninitialized urandom read (16 bytes read) [ 4.330020] systemd[1]: Initializing machine ID from random generator. [ 4.388830] random: ln: uninitialized urandom read (6 bytes read) [ 4.476479] random: systemd: uninitialized urandom read (16 bytes read) [ 4.479392] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.484641] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.490037] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.183201] device-mapper: uevent: version 1.0.3 [ 5.186138] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK [ 5.795681] random: fast init done ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.994719] virtio_net virtio0 ens2: renamed from eth0 [ 6.008576] scsi host0: ata_piix [ 6.041365] scsi host1: ata_piix [ 6.043249] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.045659] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.498548] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.724294] random: crng init done [ 10.725486] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.308125] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.607871] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.901110] SELinux: Disabled at runtime. [ 12.964668] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.977242] systemd[1]: Detected virtualization kvm. [ 12.980074] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.569755] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.573313] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.582486] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.587282] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.591173] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.600252] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.604382] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... [ 13.640596] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 14.095939] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.428724] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.446642] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.636692] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.655082] EDAC sbridge: Ver: 1.1.2 [ 16.507130] Key type dns_resolver registered [ 16.847965] NFS: Registering the id_resolver key type [ 16.850164] Key type id_resolver registered [ 16.851246] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg432-server login: [ 60.773921] libcfs: loading out-of-tree module taints kernel. [ 60.803593] Key type ._llcrypt registered [ 60.810170] Key type .llcrypt registered [ 60.934817] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 78.282802] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 79.948345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 79.958241] alg: No test for adler32 (adler32-zlib) [ 81.483765] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 82.546013] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 82.550520] LNet: Accept secure, port 988 [ 84.392164] Key type lgssc registered [ 86.841587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 91.196282] hrtimer: interrupt took 5269433 ns [ 105.908832] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 108.171641] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 116.739037] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 123.764796] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 130.337204] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 144.644544] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 156.048758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 156.102969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 156.132473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.478733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 157.541259] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 157.665367] Lustre: lustre-MDT0000: new disk, initializing [ 157.801983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.816610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 161.713792] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.755668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 173.833773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 173.909032] Lustre: 6475:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 173.950439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 173.954448] Lustre: Skipped 1 previous similar message [ 174.011284] Lustre: lustre-MDT0001: new disk, initializing [ 174.098042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 174.131733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 174.154457] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 177.637714] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.185647] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 191.246659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 191.358395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 191.588920] Lustre: lustre-OST0000: new disk, initializing [ 191.592959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 191.646065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.719941] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.674433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 197.691616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 197.786523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 209.444506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 209.521893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 209.601135] Lustre: lustre-OST0001: new disk, initializing [ 209.610805] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 209.666909] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 214.706407] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.162492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 219.169293] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 219.238937] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 225.340640] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 234.339506] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 244.711919] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing check_logdir /tmp/testlogs/ [ 248.804283] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing yml_node [ 253.937730] Lustre: DEBUG MARKER: Client: 2.16.56.105 [ 256.201252] Lustre: DEBUG MARKER: MDS: 2.16.56.105 [ 258.259344] Lustre: DEBUG MARKER: OSS: 2.16.56.105 [ 259.629833] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jul 28 18:27:54 EDT 2025 [ 276.284907] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 277.995375] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 295.912824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 295.913840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 295.935777] Lustre: Skipped 2 previous similar messages [ 295.936342] Lustre: Skipped 1 previous similar message [ 301.025977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.028682] Lustre: Skipped 2 previous similar messages [ 306.147676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.156990] Lustre: Skipped 3 previous similar messages [ 307.168189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 307.274448] LustreError: 12145:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 307.381867] Lustre: server umount lustre-MDT0000 complete [ 311.288607] LustreError: 6470:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753741726 with bad export cookie 2271203967214940705 [ 311.292540] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 311.304559] LustreError: 6470:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 316.385753] LustreError: 6484:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.390279] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.420366] LustreError: 6484:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 316.475759] Lustre: Skipped 1 previous similar message [ 316.476167] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 317.483528] LustreError: 12348:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 317.496471] LustreError: 12348:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 317.757271] Lustre: server umount lustre-MDT0001 complete [ 321.548403] LustreError: 12550:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 321.552043] LustreError: 12550:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 321.614752] Lustre: server umount lustre-OST0000 complete [ 324.798401] LustreError: 12751:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 324.804740] LustreError: 12751:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 324.982631] Lustre: server umount lustre-OST0001 complete [ 330.680583] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 339.416869] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 349.820413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 356.328821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 361.944660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 367.691207] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 373.852241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 373.932363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.166457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 374.211614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 374.271769] Lustre: lustre-MDT0000: new disk, initializing [ 374.381535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 374.415803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 378.213085] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 388.642823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 388.735111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 388.854427] Lustre: 14922:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 388.934220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 388.948613] Lustre: Skipped 1 previous similar message [ 389.052251] Lustre: lustre-MDT0001: new disk, initializing [ 389.190191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 389.249322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 389.268569] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 392.848641] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 399.227461] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 402.880357] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 409.001604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 409.102752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 409.365707] Lustre: lustre-OST0000: new disk, initializing [ 409.371793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 409.459457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.081478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 411.096839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 411.203914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 414.951909] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 422.057027] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 425.888053] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 426.120182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 429.738520] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 430.050312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 436.705162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 436.711848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 436.720663] LustreError: Skipped 1 previous similar message [ 436.721771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 436.734433] Lustre: Skipped 2 previous similar messages [ 436.745531] Lustre: Skipped 2 previous similar messages [ 437.774121] LustreError: 17830:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 437.777427] LustreError: 17830:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 437.995763] Lustre: server umount lustre-OST0000 complete [ 441.826864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.950388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 446.964649] Lustre: Skipped 3 previous similar messages [ 447.706988] LustreError: 18032:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 447.726221] LustreError: 18032:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 447.879083] Lustre: server umount lustre-MDT0000 complete [ 451.506038] LustreError: 14913:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753741866 with bad export cookie 2271203967214942238 [ 451.519337] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 451.528410] LustreError: 14913:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 451.909684] Lustre: server umount lustre-MDT0001 complete [ 456.792729] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 18:31:11 (1753741871) [ 463.200334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 463.778148] LustreError: 18761:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 463.879882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.057585] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 468.967614] LustreError: 18761:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.082818] LustreError: 18761:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.182065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 481.517230] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 486.981782] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 490.425886] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 497.023967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 502.340742] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 507.090915] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 515.673463] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 523.749983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 523.754077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.767946] Lustre: Skipped 1 previous similar message [ 523.772792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 526.907885] LustreError: 21043:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 526.924166] LustreError: 21043:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 527.113971] Lustre: server umount lustre-MDT0000 complete [ 530.408033] LustreError: 18760:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.429666] LustreError: 18760:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 530.708033] LustreError: 20938:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753741945 with bad export cookie 2271203967214943120 [ 530.710803] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 530.721746] LustreError: 20938:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 531.046408] Lustre: server umount lustre-MDT0001 complete [ 545.779687] Lustre: server umount lustre-OST0000 complete [ 565.717468] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 575.742581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 576.162536] LustreError: 22764:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.231614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 576.234016] Lustre: Skipped 2 previous similar messages [ 580.265963] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 588.805164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 593.866729] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 597.300245] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 604.533721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 610.385836] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 619.920222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 620.024652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 620.188620] Lustre: lustre-OST0001: new disk, initializing [ 620.204859] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 621.946557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 621.965536] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 622.090577] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 625.921176] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 635.316647] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 639.006306] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 655.845864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 655.851165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 655.858087] Lustre: Skipped 2 previous similar messages [ 655.866623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 655.872537] Lustre: Skipped 3 previous similar messages [ 661.942179] LustreError: 26224:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 661.979081] LustreError: 26224:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 662.166498] Lustre: server umount lustre-MDT0000 complete [ 663.012614] LustreError: 24727:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 663.048777] LustreError: 24727:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 665.789239] LustreError: 22744:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753742081 with bad export cookie 2271203967214943960 [ 665.798201] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 665.813151] LustreError: 22744:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 666.219133] Lustre: server umount lustre-MDT0001 complete [ 680.802281] Lustre: server umount lustre-OST0000 complete [ 694.793847] Lustre: server umount lustre-OST0001 complete [ 708.110143] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 716.550079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 716.983300] LustreError: 27995:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 717.093782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 717.097045] Lustre: Skipped 3 previous similar messages [ 721.544826] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 730.118142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 734.151195] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 736.745124] Lustre: 29100:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 743.398641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 750.280579] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 759.678106] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 766.066196] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 775.126834] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 778.924301] Lustre: 30944:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 788.964849] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 788.969199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 788.998360] Lustre: Skipped 6 previous similar messages [ 789.063339] Lustre: Skipped 7 previous similar messages [ 795.021355] LustreError: 31145:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 795.025405] LustreError: 31145:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 795.210901] Lustre: server umount lustre-MDT0000 complete [ 798.646931] LustreError: 27976:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753742213 with bad export cookie 2271203967214945262 [ 798.651034] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 798.664434] LustreError: 27976:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 799.062109] Lustre: server umount lustre-MDT0001 complete [ 812.825130] Lustre: server umount lustre-OST0000 complete [ 815.072271] Lustre: 3644:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753742214/real 1753742214] req@ffff8cab2f57a680 x1838931244246016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1753742230 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 816.145654] Lustre: server umount lustre-OST0001 complete [ 825.757582] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 18:37:19 (1753742239) [ 849.681631] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 858.596947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.931111] LustreError: 33717:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 858.945308] LustreError: 33717:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 862.417656] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 870.140186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 874.417109] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 877.032663] Lustre: 34821:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 882.876880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 888.456687] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 896.168161] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 901.451484] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 908.210282] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 911.256905] Lustre: 36663:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 922.084879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 922.088956] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.096322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 922.106054] Lustre: Skipped 4 previous similar messages [ 922.109652] Lustre: Skipped 6 previous similar messages [ 925.230695] LustreError: 36817:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 925.235896] LustreError: 36817:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 925.299471] Lustre: server umount lustre-MDT0000 complete [ 928.269819] LustreError: 34449:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753742343 with bad export cookie 2271203967214946368 [ 928.284794] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 928.289933] LustreError: 34449:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 928.623051] Lustre: server umount lustre-MDT0001 complete [ 941.626666] Lustre: server umount lustre-OST0000 complete [ 955.661753] Lustre: server umount lustre-OST0001 complete [ 964.623734] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 18:39:38 (1753742378) [ 980.328732] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 989.700517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.078530] LustreError: 38781:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 990.111486] LustreError: 38781:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 11 previous similar messages [ 990.182750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.186649] Lustre: Skipped 7 previous similar messages [ 993.613452] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1001.989215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1005.748644] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1008.311442] Lustre: 39884:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1014.333256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1019.347419] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1027.098793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1032.907748] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1040.447434] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1043.717959] Lustre: 41729:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1049.441657] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1049.444748] Lustre: Skipped 1 previous similar message [ 1053.159650] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.185244] Lustre: Skipped 2 previous similar messages [ 1057.936708] Lustre: server umount lustre-MDT0000 complete [ 1061.869670] LustreError: 39886:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753742477 with bad export cookie 2271203967214947474 [ 1061.875510] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1076.192181] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1076.532774] Lustre: server umount lustre-MDT0001 complete [ 1084.079685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1086.949905] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x0 to 0x1f84eff98e4e1115 [ 1086.968733] Lustre: MGC192.168.204.132@tcp: Connection restored to 192.168.204.132@tcp (at 0@lo) [ 1090.972288] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1098.574359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1102.425286] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1103.159364] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1107.423141] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1108.871332] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1111.872577] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1113.314593] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1123.266659] Lustre: server umount lustre-MDT0000 complete [ 1127.323107] LustreError: 38760:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753742542 with bad export cookie 2271203967214948629 [ 1127.324547] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1127.343102] LustreError: 38760:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1127.894193] Lustre: server umount lustre-MDT0001 complete [ 1140.920706] Lustre: server umount lustre-OST0000 complete [ 1155.501234] Lustre: server umount lustre-OST0001 complete [ 1164.236383] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 18:42:58 (1753742578) [ 1179.160463] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1188.141996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1192.176646] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1198.999689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1202.335229] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1204.751345] Lustre: 47191:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1210.057358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1215.163749] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1223.449662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1228.955984] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1236.132386] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1245.588910] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1258.412192] LustreError: 49258:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 1258.445309] LustreError: 49258:0:(obd_class.h:479:obd_check_dev()) Skipped 67 previous similar messages [ 1258.708885] Lustre: server umount lustre-MDT0000 complete [ 1261.032111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.032919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1261.043754] Lustre: Skipped 7 previous similar messages [ 1261.049032] LustreError: 46076:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.049043] LustreError: 46076:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 35 previous similar messages [ 1262.425685] LustreError: 46062:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753742677 with bad export cookie 2271203967214949441 [ 1262.436071] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1262.439122] LustreError: 46062:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1262.789579] Lustre: server umount lustre-MDT0001 complete [ 1275.915797] Lustre: server umount lustre-OST0000 complete [ 1289.776803] Lustre: server umount lustre-OST0001 complete [ 1298.496222] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 18:45:13 (1753742713) [ 1301.127877] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1302.876871] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 18:45:17 (1753742717) [ 1312.938564] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 1319.949957] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1328.434961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1334.157307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1341.577354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1341.720866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.030548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1342.232909] Lustre: lustre-MDT0000: new disk, initializing [ 1342.440810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1346.064349] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1353.552611] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1360.158793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1360.254517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1360.518357] Lustre: lustre-OST0000: new disk, initializing [ 1360.521145] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1360.527939] Lustre: Skipped 1 previous similar message [ 1361.762988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1361.775836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1361.814950] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1365.612344] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1372.244322] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1373.932529] Lustre: DEBUG MARKER: create test files [ 1402.854778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1402.860965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1402.860974] Lustre: Skipped 20 previous similar messages [ 1404.871919] Lustre: server umount lustre-OST0000 complete [ 1408.116427] Lustre: server umount lustre-MDT0000 complete [ 1417.215746] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1419.948694] Key type lgssc unregistered [ 1420.207126] LNet: 54562:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1421.224597] LNet: Removed LNI 192.168.204.132@tcp [ 1421.956339] Key type .llcrypt unregistered [ 1421.958124] Key type ._llcrypt unregistered [ 1447.388867] Key type ._llcrypt registered [ 1447.392086] Key type .llcrypt registered [ 1447.486507] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1448.619394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1448.639446] alg: No test for adler32 (adler32-zlib) [ 1449.750898] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 1450.142401] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1450.163874] LNet: Accept secure, port 988 [ 1451.897096] Key type lgssc registered [ 1453.349771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1461.201330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1461.225121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1462.992565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1468.051175] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1473.830948] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1481.017370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1481.387329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1486.220553] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1488.558868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1488.635308] Lustre: 57149:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1488.642946] Lustre: 57149:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1488.648324] Lustre: 57149:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1488.654263] Lustre: 57149:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1488.677982] Lustre: 57149:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1488.682959] Lustre: 57149:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1490.718475] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1495.470938] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1500.163481] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1534.948413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1534.951180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1535.015856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1538.040122] LustreError: 57911:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1538.182940] Lustre: server umount lustre-OST0000 complete [ 1542.589154] LustreError: 58113:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1542.621164] LustreError: 58113:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1542.876726] Lustre: server umount lustre-MDT0000 complete [ 1553.523560] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1556.454115] Key type lgssc unregistered [ 1556.716268] LNet: 58640:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.739648] LNet: Removed LNI 192.168.204.132@tcp [ 1558.612117] Key type .llcrypt unregistered [ 1558.613830] Key type ._llcrypt unregistered [ 1587.275040] Key type ._llcrypt registered [ 1587.278366] Key type .llcrypt registered [ 1587.404220] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1589.019927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1589.192414] alg: No test for adler32 (adler32-zlib) [ 1590.412074] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 1590.659601] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1590.664534] LNet: Accept secure, port 988 [ 1592.393858] Key type lgssc registered [ 1593.969548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1602.459750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1602.491457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.167817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1607.768376] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1612.826440] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1619.591181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1619.847152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1626.109384] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1629.124491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1629.310713] Lustre: 61328:0:(osd_internal.h:1335:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1629.336670] Lustre: 61328:0:(osd_handler.c:1966:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1629.356364] Lustre: 61328:0:(osd_handler.c:1973:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1629.364034] Lustre: 61328:0:(osd_handler.c:1983:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1629.385793] Lustre: 61328:0:(osd_handler.c:1990:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1629.400559] Lustre: 61328:0:(osd_handler.c:1997:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1632.781397] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1638.970561] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1649.636628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1649.645157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1649.650249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1654.099128] LustreError: 62015:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1654.354748] Lustre: server umount lustre-OST0000 complete [ 1659.319155] LustreError: 62219:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1659.322052] LustreError: 62219:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1660.160627] Lustre: server umount lustre-MDT0000 complete [ 1672.480686] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1675.247667] Key type lgssc unregistered [ 1675.500266] LNet: 62746:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.582737] LNet: Removed LNI 192.168.204.132@tcp [ 1677.399533] Key type .llcrypt unregistered [ 1677.403066] Key type ._llcrypt unregistered [ 1693.547721] Key type ._llcrypt registered [ 1693.549938] Key type .llcrypt registered [ 1693.667679] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 1707.699839] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1708.847922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1708.882289] alg: No test for adler32 (adler32-zlib) [ 1710.008756] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 1710.269826] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1710.277208] LNet: Accept secure, port 988 [ 1711.946902] Key type lgssc registered [ 1712.853133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1721.022219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1726.873646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1732.850385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1738.132235] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1743.610760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1743.640082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1743.660713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.830315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1744.859503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1744.923832] Lustre: lustre-MDT0000: new disk, initializing [ 1744.980762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1744.992243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1748.409990] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1759.129980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1759.191599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1759.273694] Lustre: 65856:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1759.358900] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1759.365958] Lustre: Skipped 1 previous similar message [ 1759.528562] Lustre: lustre-MDT0001: new disk, initializing [ 1759.666036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1759.741913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1759.773974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1763.615467] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1770.734478] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1773.893400] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1780.242151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1780.304469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1780.523100] Lustre: lustre-OST0000: new disk, initializing [ 1780.530564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1780.624159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1785.784281] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1790.021337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1790.041216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1790.197560] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1792.737262] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1795.902329] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1796.087246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1800.383960] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1800.580472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1805.796026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1805.806756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1805.825983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1808.308191] LustreError: 68763:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1808.429545] Lustre: server umount lustre-OST0000 complete [ 1816.035143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1816.035464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1816.043527] Lustre: Skipped 1 previous similar message [ 1816.048770] LustreError: Skipped 1 previous similar message [ 1816.051908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1816.056347] Lustre: Skipped 1 previous similar message [ 1818.925107] LustreError: 68965:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1818.937988] LustreError: 68965:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1819.177126] Lustre: server umount lustre-MDT0000 complete [ 1823.831590] LustreError: 65847:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753743239 with bad export cookie 3278294488874275017 [ 1823.840097] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1823.859591] LustreError: 65847:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1824.097316] LustreError: 69167:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1824.105120] LustreError: 69167:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1824.508371] Lustre: server umount lustre-MDT0001 complete [ 1837.612639] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 18:54:11 (1753743251) [ 1853.768302] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1868.093230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1888.358720] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 1894.957661] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1902.821940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1908.007345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1913.093362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1918.287336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1922.887826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1922.935965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1923.110208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1923.141037] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1923.210371] Lustre: lustre-MDT0000: new disk, initializing [ 1923.304478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1923.335394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1926.072334] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1935.088656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1935.152120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1935.224439] Lustre: 73403:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1935.276766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1935.279743] Lustre: Skipped 1 previous similar message [ 1935.329201] Lustre: lustre-MDT0001: new disk, initializing [ 1935.446828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1935.493550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1935.518042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1938.786386] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1945.911304] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1949.373851] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1957.656932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1957.705749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1957.911830] Lustre: lustre-OST0000: new disk, initializing [ 1957.914312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1957.984713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1962.496204] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1966.626570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1966.637585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1966.709801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1968.686740] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1972.376769] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1972.616841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1976.101506] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1976.344148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1981.922969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1981.931244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1981.936683] Lustre: Skipped 2 previous similar messages [ 1981.939899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1984.185177] LustreError: 76309:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1984.192728] LustreError: 76309:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1984.423449] Lustre: server umount lustre-OST0000 complete [ 1991.649647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1991.653132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1991.671964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1991.674967] Lustre: Skipped 1 previous similar message [ 1997.286691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1997.291216] Lustre: Skipped 3 previous similar messages [ 2002.400172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2002.475735] LustreError: 76511:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2002.484361] LustreError: 76511:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2002.565482] Lustre: server umount lustre-MDT0000 complete [ 2005.423964] LustreError: 76512:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753743420 with bad export cookie 3278294488874275892 [ 2005.428152] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2005.440935] LustreError: 76512:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2005.765333] Lustre: server umount lustre-MDT0001 complete [ 2012.613236] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 18:57:07 (1753743427) [ 2018.857395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2019.543585] LustreError: 77385:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2019.564423] LustreError: 77385:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2019.615972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.993280] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2024.936138] LustreError: 77385:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2029.030446] LustreError: 77385:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2031.659210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2035.183125] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2040.395719] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2043.453681] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2049.388931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2054.827387] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2059.691282] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2062.790378] Lustre: 79705:0:(genops.c:1791:obd_export_evict_by_uuid()) MGS: evicting a9e55fe6-f5fb-49a5-9d28-aae1e4f66f56 at adminstrative request [ 2063.910214] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2063.928847] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2074.086675] Lustre: MGS: Client a9e55fe6-f5fb-49a5-9d28-aae1e4f66f56 (at 0@lo) reconnecting [ 2074.112050] Lustre: Evicted from MGS (at 192.168.204.132@tcp) after server handle changed from 0x2d7ed777ee78d7ad to 0x2d7ed777ee78da4d [ 2074.128935] Lustre: MGC192.168.204.132@tcp: Connection restored to 192.168.204.132@tcp (at 0@lo) [ 2100.242078] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2100.330450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2100.485876] Lustre: lustre-OST0001: new disk, initializing [ 2100.494060] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2100.581113] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2100.584587] Lustre: Skipped 2 previous similar messages [ 2106.920262] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2106.965262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2106.982651] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2107.125169] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2113.707525] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2117.601213] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2117.609409] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.611388] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2117.611396] Lustre: Skipped 2 previous similar messages [ 2117.658406] Lustre: Skipped 2 previous similar messages [ 2122.763461] LustreError: 80836:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2122.773852] LustreError: 80836:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2122.895077] Lustre: server umount lustre-OST0001 complete [ 2127.844274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.847361] LustreError: 78971:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2127.866408] Lustre: Skipped 1 previous similar message [ 2127.871225] LustreError: 78971:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2132.690152] Lustre: server umount lustre-OST0000 complete [ 2138.091499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2138.094350] Lustre: Skipped 5 previous similar messages [ 2142.759230] LustreError: 81239:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2142.764644] LustreError: 81239:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2142.877329] Lustre: server umount lustre-MDT0000 complete [ 2143.210570] LustreError: 77385:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2143.225414] LustreError: 77385:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2146.384790] LustreError: 77364:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753743561 with bad export cookie 3278294488874277453 [ 2146.397068] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2146.425956] LustreError: 77364:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2146.810739] Lustre: server umount lustre-MDT0001 complete [ 2154.463384] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 18:59:28 (1753743568) [ 2156.155775] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2157.851911] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 18:59:32 (1753743572) [ 2159.514856] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2161.335647] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 18:59:35 (1753743575) [ 2162.895957] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2164.455458] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 18:59:39 (1753743579) [ 2165.898604] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2170.317590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2171.255916] loop: module loaded [ 2171.267941] loop0: detected capacity change from 0 to 4294967296 [ 2171.323231] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2172.750651] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2175.154543] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2177.860263] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2185.556024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2188.886873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2192.783430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2196.555188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2201.895154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2201.954052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2201.976455] LustreError: 84008:0:(obd_config.c:779:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2201.982526] LustreError: 84008:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2201.989844] LustreError: 84008:0:(tgt_mount.c:2240:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2201.994369] LustreError: 84008:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2217.157901] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 2224.056273] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2232.424828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2238.635988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2244.196365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2250.872221] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2256.495112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2256.557059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.809337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2256.836505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2256.948690] Lustre: lustre-MDT0000: new disk, initializing [ 2257.074502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2257.099173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2260.563895] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2270.590480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2270.669973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2270.746878] Lustre: 86849:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2271.020379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2274.704469] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2282.621352] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2285.966890] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2291.759938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2291.830770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2297.676468] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2301.486938] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2301.580163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2304.488880] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2307.738771] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2307.948160] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2311.821784] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2311.958452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2316.770591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.771502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2316.786315] Lustre: Skipped 2 previous similar messages [ 2316.790536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2316.802325] Lustre: Skipped 1 previous similar message [ 2319.733136] LustreError: 89756:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2319.754930] LustreError: 89756:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2319.983135] Lustre: server umount lustre-OST0000 complete [ 2338.272340] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2338.512125] Lustre: server umount lustre-MDT0000 complete [ 2341.488619] LustreError: 87632:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753743756 with bad export cookie 3278294488874278048 [ 2341.489309] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2341.504510] LustreError: 87632:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2341.921500] Lustre: server umount lustre-MDT0001 complete [ 2350.266905] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 19:02:44 (1753743764) [ 2356.754734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2357.115237] LustreError: 90834:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2360.798540] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2368.930810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2372.980634] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2378.425889] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2382.346123] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2388.836329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2389.142652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.148602] Lustre: Skipped 4 previous similar messages [ 2394.662477] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2399.739917] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2403.563671] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2403.775861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2407.254575] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2407.416703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2413.756932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2413.827804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2413.981531] Lustre: lustre-OST0001: new disk, initializing [ 2413.987301] Lustre: Skipped 2 previous similar messages [ 2413.991777] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2413.994636] Lustre: Skipped 3 previous similar messages [ 2418.736807] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2419.816302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2419.826709] Lustre: Skipped 2 previous similar messages [ 2419.829770] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2419.910762] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2424.632560] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2427.715558] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2427.891678] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2431.393756] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2431.605342] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2447.750472] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2469.251176] Lustre: Failing over lustre-MDT0000 [ 2469.566599] LustreError: 95691:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2469.574978] LustreError: 95691:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2469.620773] LustreError: 90830:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.644636] LustreError: 90830:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2469.710293] Lustre: server umount lustre-MDT0000 complete [ 2471.407697] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.425198] Lustre: Skipped 5 previous similar messages [ 2486.752955] Lustre: 64072:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753743886/real 1753743886] req@00000000b527a811 x1838932951288192/t0(0) o400->MGC192.168.204.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1753743902 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2486.767069] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2493.834024] LDISKFS-fs (dm-0): recovery complete [ 2493.836684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2497.010686] LustreError: 64070:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008c69675b x1838932951295488/t0(0) o250->MGC192.168.204.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2497.373047] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2501.281624] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2501.344667] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2501.345088] Lustre: 92040:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2502.633913] Lustre: 90834:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2502.634498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2502.638441] Lustre: 90834:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 1 previous similar message [ 2502.721094] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2503.048135] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2503.373951] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2503.385049] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 1 previous similar message [ 2503.704242] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2503.710137] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 1 previous similar message [ 2503.721975] Lustre: 96440:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2503.731445] Lustre: 96440:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 7 previous similar messages [ 2504.686165] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2504.699725] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 3 previous similar messages [ 2505.017067] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2505.033328] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 3 previous similar messages [ 2506.002717] Lustre: 96440:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2506.025269] Lustre: 96440:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 6 previous similar messages [ 2506.977377] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2506.991893] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 6 previous similar messages [ 2507.312945] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2507.319578] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 6 previous similar messages [ 2510.161300] Lustre: 96440:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2510.182793] Lustre: 96440:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 12 previous similar messages [ 2511.131878] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2511.141360] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 12 previous similar messages [ 2511.456153] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2511.459264] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 12 previous similar messages [ 2518.441079] Lustre: 96440:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2518.456451] Lustre: 96440:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 25 previous similar messages [ 2519.407430] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2519.412561] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 25 previous similar messages [ 2519.736190] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2519.748490] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 25 previous similar messages [ 2534.638812] Lustre: 96440:0:(ldlm_lib.c:2001:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2534.658138] Lustre: 96440:0:(ldlm_lib.c:2001:extend_recovery_timer()) Skipped 50 previous similar messages [ 2535.610802] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2535.618862] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 50 previous similar messages [ 2535.944139] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2535.955293] LustreError: 96440:0:(ldlm_lib.c:2613:replay_request_or_update()) Skipped 50 previous similar messages [ 2561.080258] Lustre: 96440:0:(ldlm_lib.c:2309:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2561.087662] Lustre: 96440:0:(ldlm_lib.c:2319:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2561.098208] Lustre: 96440:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 71c5c7bf-1501-46b1-ac28-6f3207c5d73c@192.168.204.32@tcp [ 2561.108520] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2561.113846] Lustre: 96440:0:(ldlm_lib.c:1831:abort_req_replay_queue()) @@@ aborted: req@000000000302c637 x1838932937479296/t0(8589934780) o35->71c5c7bf-1501-46b1-ac28-6f3207c5d73c@192.168.204.32@tcp:542/0 lens 392/0 e 0 to 0 dl 1753743987 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2561.141786] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2561.143792] Lustre: lustre-MDT0000: Denying connection for new client 71c5c7bf-1501-46b1-ac28-6f3207c5d73c (at 192.168.204.32@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 2561.164305] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2561.190714] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 2561.196710] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2561.202949] Lustre: Skipped 2 previous similar messages [ 2561.239839] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 2561.241152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 2572.257213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2572.262100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.271086] Lustre: Skipped 1 previous similar message [ 2572.277629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2572.288399] Lustre: Skipped 6 previous similar messages [ 2577.097700] Lustre: server umount lustre-OST0000 complete [ 2579.436196] LustreError: 92422:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.447737] LustreError: 92422:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 42 previous similar messages [ 2586.757192] Lustre: server umount lustre-OST0001 complete [ 2595.881486] Lustre: server umount lustre-MDT0000 complete [ 2598.803406] LustreError: 97505:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744014 with bad export cookie 3278294488874329267 [ 2598.817433] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2598.818765] LustreError: 97505:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2598.940480] LustreError: 97706:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2598.949564] LustreError: 97706:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 2599.170976] Lustre: server umount lustre-MDT0001 complete [ 2606.000579] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 19:07:00 (1753744020) [ 2612.146962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2612.151514] Lustre: *** cfs_fail_loc=197, val=0*** [ 2629.731638] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2635.872585] LustreError: 98398:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.132@tcp: failed processing log, type 4: rc = -110 [ 2642.054305] Lustre: *** cfs_fail_loc=197, val=0*** [ 2642.057241] Lustre: Skipped 65 previous similar messages [ 2665.700535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2665.708702] Lustre: Skipped 2 previous similar messages [ 2670.603901] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2675.440744] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2677.094636] Lustre: server umount lustre-OST0000 complete [ 2683.300649] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 19:08:18 (1753744098) [ 2701.478593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2701.770183] LustreError: 99899:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2701.789275] LustreError: 99899:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2704.536601] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2709.805226] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2712.851651] LustreError: 99930:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2712.857070] LustreError: 99930:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2712.861110] LustreError: 99930:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 2713.153275] Lustre: server umount lustre-MDT0000 complete [ 2723.356468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2735.572919] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 2742.536515] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2750.121172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2755.072127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2759.392953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2764.501926] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2769.460851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2769.503807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2769.659557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2769.683221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2769.729917] Lustre: lustre-MDT0000: new disk, initializing [ 2769.825995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2772.628683] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2780.643714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2780.703675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2780.786423] Lustre: 103947:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2780.992885] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2783.749360] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2789.628912] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2792.539451] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2797.161578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2797.216528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2801.284415] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2806.925343] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2807.114726] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2809.693425] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2809.842388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2812.366748] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2812.562147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2816.998558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2817.002498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2817.015699] LustreError: Skipped 3 previous similar messages [ 2817.021532] Lustre: Skipped 6 previous similar messages [ 2817.023792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2817.025678] Lustre: Skipped 11 previous similar messages [ 2828.260477] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2828.398784] Lustre: server umount lustre-OST0000 complete [ 2837.848810] Lustre: server umount lustre-MDT0000 complete [ 2840.191428] LustreError: 103939:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744255 with bad export cookie 3278294488874339935 [ 2840.196117] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2840.196646] LustreError: 103939:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2840.489681] Lustre: server umount lustre-MDT0001 complete [ 2846.402741] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 19:11:01 (1753744261) [ 2853.423441] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 2855.496626] Key type lgssc unregistered [ 2855.755219] LNet: 107981:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2856.810715] LNet: Removed LNI 192.168.204.132@tcp [ 2857.424156] Key type .llcrypt unregistered [ 2857.426527] Key type ._llcrypt unregistered [ 2870.799317] Key type ._llcrypt registered [ 2870.803399] Key type .llcrypt registered [ 2870.882955] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 2882.165268] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2882.694270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2882.724091] alg: No test for adler32 (adler32-zlib) [ 2883.737633] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 2883.953115] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 2883.955948] LNet: Accept secure, port 988 [ 2885.624176] Key type lgssc registered [ 2886.438713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2894.319248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2899.466517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2904.166129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2909.300184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2913.574972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2917.932639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2917.975877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2917.988114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2919.154485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2919.184139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2919.246942] Lustre: lustre-MDT0000: new disk, initializing [ 2919.324318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2919.344176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2921.882416] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2929.358179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2929.404301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2929.570249] Lustre: lustre-OST0000: new disk, initializing [ 2929.575655] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2929.578450] Lustre: Skipped 1 previous similar message [ 2929.632039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2933.728592] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2937.983768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2937.998551] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2938.058977] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2942.739201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2942.810877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2942.970811] Lustre: lustre-OST0001: new disk, initializing [ 2942.975543] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2943.042061] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2946.458468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 2946.464299] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 2946.501617] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 2947.215628] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2956.773896] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.778512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2956.779257] Lustre: Skipped 1 previous similar message [ 2959.964563] LustreError: 113503:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 2960.105273] Lustre: server umount lustre-MDT0000 complete [ 2964.446424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2967.611929] LustreError: 111282:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744382 with bad export cookie 16918875646100147832 [ 2967.615379] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2967.622128] LustreError: 111282:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2967.655142] LustreError: 114097:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2967.662240] LustreError: 114097:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2967.689486] Lustre: server umount lustre-OST0000 complete [ 2970.477381] LustreError: 114298:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2970.480770] LustreError: 114298:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2970.649141] Lustre: server umount lustre-OST0001 complete [ 2977.114876] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 19:13:11 (1753744391) [ 2984.708382] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2990.974460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2991.755599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2995.657171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2996.424365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3001.282581] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 19:13:36 (1753744416) [ 3010.806780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3012.250542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3013.581411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3014.776330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3015.421037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3016.656415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3029.894842] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 3037.182801] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3044.984243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3051.191814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3057.158216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3063.638380] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3071.010438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3071.077466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3071.463877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3071.530310] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3071.658594] Lustre: lustre-MDT0000: new disk, initializing [ 3071.762751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3071.782539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3076.117122] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3086.739574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3086.810606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3086.889588] Lustre: 119597:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3086.931802] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3086.934626] Lustre: Skipped 1 previous similar message [ 3086.988430] Lustre: lustre-MDT0001: new disk, initializing [ 3087.128826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3087.190430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3087.204996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3091.445916] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3099.535980] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3103.766795] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3110.313995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3110.394412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3110.672632] Lustre: lustre-OST0000: new disk, initializing [ 3110.677461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3110.784446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3116.375744] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3117.632815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3117.646404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3117.735756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3123.142126] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3126.636500] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3126.832094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3130.241603] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3130.445544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3133.415314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.417384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3133.433528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3133.433536] Lustre: Skipped 1 previous similar message [ 3138.100201] LustreError: 122506:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3138.103295] LustreError: 122506:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3138.250663] Lustre: server umount lustre-OST0000 complete [ 3143.649224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3143.655019] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.658411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3143.671328] Lustre: Skipped 2 previous similar messages [ 3143.676647] Lustre: Skipped 2 previous similar messages [ 3147.303318] LustreError: 122708:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3147.307298] LustreError: 122708:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3147.361454] Lustre: server umount lustre-MDT0000 complete [ 3150.229044] LustreError: 119590:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744565 with bad export cookie 16918875646100149057 [ 3150.240915] LustreError: 119590:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3150.242916] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3150.634419] Lustre: server umount lustre-MDT0001 complete [ 3157.767734] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 19:16:12 (1753744572) [ 3163.291892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.575139] LustreError: 123583:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3163.591792] LustreError: 123583:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3163.642881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3166.603819] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3168.738606] LustreError: 123583:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3172.836491] LustreError: 123583:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.689588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3176.784539] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3180.943469] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3183.647989] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3189.228449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3194.129175] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3197.941851] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3212.827257] Lustre: *** cfs_fail_loc=159, val=0*** [ 3228.456481] Lustre: lustre-MDT0000: Client 2d184519-a3e6-4feb-bd1e-a0d6a362ec8e (at 192.168.204.32@tcp) reconnecting [ 3229.946288] Lustre: *** cfs_fail_loc=159, val=0*** [ 3229.960472] Lustre: Skipped 5 previous similar messages [ 3245.326494] Lustre: lustre-MDT0000: Client 2d184519-a3e6-4feb-bd1e-a0d6a362ec8e (at 192.168.204.32@tcp) reconnecting [ 3250.148483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3250.163326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3250.186690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3251.391343] LustreError: 126070:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3251.403046] LustreError: 126070:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3251.569863] Lustre: server umount lustre-OST0000 complete [ 3255.777808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3255.784313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.804665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3264.488260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3264.490583] Lustre: Skipped 2 previous similar messages [ 3269.088144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3269.172276] LustreError: 126270:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3269.175338] LustreError: 126270:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3269.242210] Lustre: server umount lustre-MDT0000 complete [ 3269.626875] LustreError: 123583:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3269.641206] LustreError: 123583:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3272.189200] LustreError: 123564:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744687 with bad export cookie 16918875646100149939 [ 3272.205932] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3272.628357] Lustre: server umount lustre-MDT0001 complete [ 3282.931425] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 3285.745542] Key type lgssc unregistered [ 3286.024617] LNet: 127000:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3287.084779] LNet: Removed LNI 192.168.204.132@tcp [ 3287.967119] Key type .llcrypt unregistered [ 3287.970109] Key type ._llcrypt unregistered [ 3309.812211] Key type ._llcrypt registered [ 3309.813750] Key type .llcrypt registered [ 3309.889375] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3310.571870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3310.697331] alg: No test for adler32 (adler32-zlib) [ 3311.780975] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 3311.978946] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 3311.984541] LNet: Accept secure, port 988 [ 3313.712224] Key type lgssc registered [ 3314.610908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3321.792761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3321.825873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.215155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3326.658849] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3333.603867] LustreError: 128384:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.649880] LustreError: 128384:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3336.080965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3336.676387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3339.929487] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3345.293753] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3349.627149] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3356.487488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3356.971502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3362.758403] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3367.640598] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3368.252915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3377.611314] Lustre: *** cfs_fail_loc=159, val=0*** [ 3393.798976] Lustre: lustre-MDT0000: Client 859b6cfc-ed66-40a3-a244-f5d592c19845 (at 192.168.204.32@tcp) reconnecting [ 3396.335291] Lustre: *** cfs_fail_loc=159, val=0*** [ 3412.762489] Lustre: lustre-MDT0001: Client 859b6cfc-ed66-40a3-a244-f5d592c19845 (at 192.168.204.32@tcp) reconnecting [ 3413.994891] Lustre: *** cfs_fail_loc=159, val=0*** [ 3413.996656] Lustre: Skipped 3 previous similar messages [ 3429.706721] Lustre: lustre-MDT0001: Client 859b6cfc-ed66-40a3-a244-f5d592c19845 (at 192.168.204.32@tcp) reconnecting [ 3434.283923] Lustre: 131073:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3436.983341] Lustre: *** cfs_fail_loc=159, val=0*** [ 3436.990205] Lustre: Skipped 6 previous similar messages [ 3452.162888] Lustre: lustre-MDT0000: Client 9095fa64-9775-41c5-a9a5-c6a76299a6b1 (at 192.168.204.32@tcp) reconnecting [ 3453.228482] Lustre: *** cfs_fail_loc=159, val=0*** [ 3453.230363] Lustre: Skipped 12 previous similar messages [ 3469.609087] Lustre: lustre-MDT0000: Client 9095fa64-9775-41c5-a9a5-c6a76299a6b1 (at 192.168.204.32@tcp) reconnecting [ 3470.885629] Lustre: 131363:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3474.931689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3474.939602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3474.967172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3475.943419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3475.946480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3481.063450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3481.072715] Lustre: Skipped 2 previous similar messages [ 3486.179933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3486.185433] Lustre: Skipped 1 previous similar message [ 3489.248246] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3489.293609] LustreError: 131460:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3489.465415] Lustre: server umount lustre-OST0000 complete [ 3495.393292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3495.397711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3495.403839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3498.727557] LustreError: 131664:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3498.733636] LustreError: 131664:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3498.840900] Lustre: server umount lustre-MDT0000 complete [ 3501.542478] LustreError: 129091:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3501.550893] LustreError: 129091:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3501.880413] LustreError: 131194:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753744917 with bad export cookie 18384903890881369425 [ 3501.882564] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3501.896961] LustreError: 131194:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3502.115049] LustreError: 131867:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3502.118525] LustreError: 131867:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3502.322574] Lustre: server umount lustre-MDT0001 complete [ 3512.582499] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 3515.059480] Key type lgssc unregistered [ 3515.349297] LNet: 132394:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3516.392650] LNet: Removed LNI 192.168.204.132@tcp [ 3517.208067] Key type .llcrypt unregistered [ 3517.209910] Key type ._llcrypt unregistered [ 3534.169759] Key type ._llcrypt registered [ 3534.171668] Key type .llcrypt registered [ 3534.245265] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3535.016746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3535.082914] alg: No test for adler32 (adler32-zlib) [ 3536.095884] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 3536.301098] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 3536.304253] LNet: Accept secure, port 988 [ 3537.976189] Key type lgssc registered [ 3538.871660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3544.973357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3545.012678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3546.419597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3549.585230] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3555.814482] LustreError: 133776:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3555.835690] LustreError: 133776:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3556.985404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3557.266348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3560.282887] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3564.629082] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3567.811507] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3573.443995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3573.814801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3578.909600] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3582.055787] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 3583.241635] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3591.251158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 3599.814652] Lustre: 136012:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3601.961231] Lustre: 136060:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3603.368842] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 3604.432789] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 3607.849119] Lustre: 136207:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 3608.866308] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 3609.945454] Lustre: 136305:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3616.750164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3616.753215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3616.764483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3616.798429] Lustre: Skipped 1 previous similar message [ 3620.160253] LustreError: 136413:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3620.450697] Lustre: server umount lustre-OST0000 complete [ 3626.980969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3626.984829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3627.004588] Lustre: Skipped 1 previous similar message [ 3627.025425] Lustre: Skipped 2 previous similar messages [ 3630.354826] LustreError: 136614:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3630.370210] LustreError: 136614:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3630.500951] Lustre: server umount lustre-MDT0000 complete [ 3632.103850] LustreError: 133757:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3632.116713] LustreError: 133757:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3633.754494] LustreError: 133737:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753745049 with bad export cookie 8710355920601830808 [ 3633.761065] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3633.762271] LustreError: 133737:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3633.894794] LustreError: 136816:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3633.900897] LustreError: 136816:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3634.182799] Lustre: server umount lustre-MDT0001 complete [ 3645.794649] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 3648.574530] Key type lgssc unregistered [ 3648.871527] LNet: 137344:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3649.915589] LNet: Removed LNI 192.168.204.132@tcp [ 3650.715947] Key type .llcrypt unregistered [ 3650.717747] Key type ._llcrypt unregistered [ 3671.916304] Key type ._llcrypt registered [ 3671.918916] Key type .llcrypt registered [ 3672.016385] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3673.173518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3673.201396] alg: No test for adler32 (adler32-zlib) [ 3674.367480] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 3674.629929] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 3674.632166] LNet: Accept secure, port 988 [ 3676.304182] Key type lgssc registered [ 3677.240794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3685.527089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3685.547585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3687.168752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3690.472582] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3697.635159] LustreError: 138728:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3697.667485] LustreError: 138728:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3699.505505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3699.866640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3704.425822] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3709.730913] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3713.535302] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3720.358441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3720.839877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3724.921491] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 3726.292651] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3731.446324] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3732.081227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 3737.505838] Lustre: *** cfs_fail_loc=159, val=0*** [ 3753.747877] Lustre: lustre-MDT0000: Client 78dcf01f-0a20-4e7a-8439-b7e01582a74a (at 192.168.204.32@tcp) reconnecting [ 3760.101546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3760.104488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3760.125689] LustreError: Skipped 1 previous similar message [ 3760.168678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3761.122072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3761.125562] LustreError: Skipped 6 previous similar messages [ 3761.151153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3761.184710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3763.172285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3763.179915] Lustre: Skipped 1 previous similar message [ 3764.930444] LustreError: 141090:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3765.101881] Lustre: server umount lustre-OST0000 complete [ 3771.363023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3771.378010] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.400872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3773.411421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3778.539186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3778.551592] Lustre: Skipped 3 previous similar messages [ 3783.136555] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3783.212020] LustreError: 141291:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3783.223955] LustreError: 141291:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3783.321634] Lustre: server umount lustre-MDT0000 complete [ 3783.649194] LustreError: 139439:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3783.659067] LustreError: 139439:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3787.130610] LustreError: 138709:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753745202 with bad export cookie 14600527282299108735 [ 3787.136058] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3787.165064] LustreError: 138709:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3787.350035] LustreError: 141495:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3787.353759] LustreError: 141495:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3787.624559] Lustre: server umount lustre-MDT0001 complete [ 3798.526354] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 3801.650442] Key type lgssc unregistered [ 3801.891797] LNet: 142023:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3802.918902] LNet: Removed LNI 192.168.204.132@tcp [ 3803.898513] Key type .llcrypt unregistered [ 3803.907860] Key type ._llcrypt unregistered [ 3826.392279] Key type ._llcrypt registered [ 3826.393806] Key type .llcrypt registered [ 3826.513364] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3827.540194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3827.748214] alg: No test for adler32 (adler32-zlib) [ 3828.956214] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 3829.282834] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 3829.287605] LNet: Accept secure, port 988 [ 3830.968159] Key type lgssc registered [ 3831.899709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3839.142671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3839.156537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3840.568290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3843.888743] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3850.701723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3850.723846] LustreError: 143408:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3850.737697] LustreError: 143408:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3850.974234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3853.835870] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3858.037388] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3861.024660] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3867.641625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3867.930818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3872.686401] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3876.147996] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 3877.081426] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3886.394527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 3905.239568] Lustre: lustre-OST0000: evicting 8532cf92-0c62-46fe-ae22-5b0771828545 (at 192.168.204.32@tcp) by administrative request [ 3937.772455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3937.776348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.787355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3937.797099] Lustre: Skipped 1 previous similar message [ 3937.805454] Lustre: Skipped 1 previous similar message [ 3939.957962] LustreError: 146194:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3940.118554] Lustre: server umount lustre-OST0000 complete [ 3948.006098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3948.020231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3948.031758] Lustre: Skipped 1 previous similar message [ 3949.801224] LustreError: 146397:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3949.805511] LustreError: 146397:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3949.951253] Lustre: server umount lustre-MDT0000 complete [ 3953.121424] LustreError: 145560:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3953.145162] LustreError: 145560:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3953.556873] LustreError: 143387:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753745368 with bad export cookie 6622806732569878477 [ 3953.560591] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3953.574742] LustreError: 143387:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3953.791213] LustreError: 146598:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3953.807990] LustreError: 146598:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3954.143655] Lustre: server umount lustre-MDT0001 complete [ 3965.425376] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 3968.139565] Key type lgssc unregistered [ 3968.393776] LNet: 147125:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3969.448755] LNet: Removed LNI 192.168.204.132@tcp [ 3970.217994] Key type .llcrypt unregistered [ 3970.222429] Key type ._llcrypt unregistered [ 4041.705334] Key type ._llcrypt registered [ 4041.706736] Key type .llcrypt registered [ 4041.794984] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4042.741742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4042.793674] alg: No test for adler32 (adler32-zlib) [ 4044.072608] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 4044.386723] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4044.389712] LNet: Accept secure, port 988 [ 4046.072648] Key type lgssc registered [ 4047.360644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4055.788349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4055.843685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4057.499852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4061.452412] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4066.788158] LustreError: 149582:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4066.816805] LustreError: 149582:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4069.739740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4070.589023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4075.041197] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4080.013421] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4083.738293] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4089.956980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4090.433575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4095.406855] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4095.535618] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4100.454285] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4102.713356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4115.941985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4115.955924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.977033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4117.985611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4117.993829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4117.998935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4119.166533] LustreError: 151833:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4119.311260] Lustre: server umount lustre-OST0000 complete [ 4123.105549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4123.113171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4123.116079] Lustre: Skipped 1 previous similar message [ 4123.128641] Lustre: Skipped 2 previous similar messages [ 4128.228569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4128.834378] LustreError: 152036:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4128.848996] LustreError: 152036:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4128.917459] Lustre: server umount lustre-MDT0000 complete [ 4132.314855] LustreError: 152037:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753745547 with bad export cookie 16883454332889596984 [ 4132.320468] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4132.327579] LustreError: 152037:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4132.474495] LustreError: 152240:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4132.481146] LustreError: 152240:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4132.721479] Lustre: server umount lustre-MDT0001 complete [ 4143.271516] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4145.957743] Key type lgssc unregistered [ 4146.288269] LNet: 152767:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4147.370874] LNet: Removed LNI 192.168.204.132@tcp [ 4148.139612] Key type .llcrypt unregistered [ 4148.141346] Key type ._llcrypt unregistered [ 4168.556316] Key type ._llcrypt registered [ 4168.557950] Key type .llcrypt registered [ 4168.656840] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4169.565482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4169.673035] alg: No test for adler32 (adler32-zlib) [ 4170.750870] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 4170.931775] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4170.937240] LNet: Accept secure, port 988 [ 4172.600273] Key type lgssc registered [ 4173.669897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4180.010150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4186.920792] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 19:33:21 (1753745601) [ 4202.108724] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 4211.479439] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4220.140819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4225.930131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4231.850506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4237.543591] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4243.276532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4243.324267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4243.340433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4244.643702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4244.693293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4244.776769] Lustre: lustre-MDT0000: new disk, initializing [ 4244.928184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4244.956553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4248.437843] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4257.371465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4257.428086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4257.513966] Lustre: 156952:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4257.556594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4257.559137] Lustre: Skipped 1 previous similar message [ 4257.613419] Lustre: lustre-MDT0001: new disk, initializing [ 4257.714724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4257.767575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4257.772340] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4260.737989] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4267.270391] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4270.115810] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4275.810769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4275.901660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4276.095743] Lustre: lustre-OST0000: new disk, initializing [ 4276.105204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4276.182833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4281.764762] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4282.414402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4282.427655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4282.530248] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4288.339224] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4297.530153] LustreError: 159521:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4297.754400] Lustre: server umount lustre-OST0000 complete [ 4303.339994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4303.362402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4308.453080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4308.459425] Lustre: Skipped 2 previous similar messages [ 4311.537477] LustreError: 159721:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4311.550041] LustreError: 159721:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4311.633874] Lustre: server umount lustre-MDT0000 complete [ 4314.593980] Lustre: 153457:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1753745713/real 1753745713] req@0000000085635151 x1838935531619200/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1753745729 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4314.594123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4314.629363] Lustre: 153457:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4314.645856] Lustre: Skipped 1 previous similar message [ 4315.480355] LustreError: 156945:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753745730 with bad export cookie 15377435968613635041 [ 4315.486627] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4315.493643] LustreError: 156945:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4315.650834] LustreError: 159923:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4315.653786] LustreError: 159923:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4316.011121] Lustre: server umount lustre-MDT0001 complete [ 4326.420954] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4328.987411] Key type lgssc unregistered [ 4329.265525] LNet: 160450:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4330.342901] LNet: Removed LNI 192.168.204.132@tcp [ 4331.139988] Key type .llcrypt unregistered [ 4331.144109] Key type ._llcrypt unregistered [ 4351.882650] Key type ._llcrypt registered [ 4351.884116] Key type .llcrypt registered [ 4351.976737] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4352.874085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4352.901444] alg: No test for adler32 (adler32-zlib) [ 4353.948192] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 4354.169695] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4354.174092] LNet: Accept secure, port 988 [ 4355.834977] Key type lgssc registered [ 4356.721841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4364.331791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4364.365116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4365.954402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4369.067880] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4376.555173] LustreError: 161843:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4376.571313] LustreError: 161843:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4376.684448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4376.945663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4380.593415] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4385.335857] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4388.482647] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4394.619310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4394.932443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4399.624275] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4404.177992] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4411.315331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4420.020241] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4420.037422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4420.044812] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4420.051241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4420.061200] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.132@tcp (at 0@lo) [ 4420.864458] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4420.877251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4420.891417] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4420.904381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4420.921054] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.132@tcp (at 0@lo) [ 4421.777160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4422.569702] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4422.579671] Lustre: Skipped 1 previous similar message [ 4422.590959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4422.604716] Lustre: Skipped 1 previous similar message [ 4422.613930] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4422.620698] Lustre: Skipped 1 previous similar message [ 4422.634147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4422.652235] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.132@tcp (at 0@lo) [ 4422.659849] Lustre: Skipped 1 previous similar message [ 4423.337560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4424.068771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4424.881779] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4424.894333] Lustre: Skipped 2 previous similar messages [ 4424.908103] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4424.925728] Lustre: Skipped 2 previous similar messages [ 4424.929594] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4424.932413] Lustre: Skipped 2 previous similar messages [ 4424.936266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4424.946336] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.132@tcp (at 0@lo) [ 4424.950184] Lustre: Skipped 2 previous similar messages [ 4425.661170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4426.546491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4427.322412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4428.119241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4428.779504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4429.455723] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4429.459291] Lustre: Skipped 5 previous similar messages [ 4429.472420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4429.478142] Lustre: Skipped 5 previous similar messages [ 4429.482485] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4429.485652] Lustre: Skipped 5 previous similar messages [ 4429.489683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4429.495621] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.132@tcp (at 0@lo) [ 4429.498686] Lustre: Skipped 5 previous similar messages [ 4430.220796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4430.890528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4431.655759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4432.380294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4433.141447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4433.793715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4434.520836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4435.247139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4436.081098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4436.983540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4437.916970] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4437.922844] Lustre: Skipped 10 previous similar messages [ 4437.956437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4437.962233] Lustre: Skipped 10 previous similar messages [ 4437.964723] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4437.968871] Lustre: Skipped 10 previous similar messages [ 4437.974321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4437.979615] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.132@tcp (at 0@lo) [ 4437.983204] Lustre: Skipped 10 previous similar messages [ 4438.868195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4439.722723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4440.654394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4441.440039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4442.361355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4443.173057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4443.983589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4444.721368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4445.609334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4446.383507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4447.242408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4448.121966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4448.913381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4449.660562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4450.319241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4451.127056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4452.132280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4452.882930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4453.569247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4454.244507] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4454.250249] Lustre: Skipped 19 previous similar messages [ 4454.258953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.263209] Lustre: Skipped 19 previous similar messages [ 4454.273468] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4454.276414] Lustre: Skipped 19 previous similar messages [ 4454.278569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4454.282030] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.132@tcp (at 0@lo) [ 4454.284907] Lustre: Skipped 19 previous similar messages [ 4455.041678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4456.066906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4456.896473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4457.672142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4458.422692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4459.179976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4460.002582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4460.749835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4461.580096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4462.379583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4463.296762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4464.195911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4465.213175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4466.017770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4466.951391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4467.699928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4468.719252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4469.501660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4470.285751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4471.167613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4471.956272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4472.654172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4473.428126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4474.175099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4474.893494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4475.553369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4476.327131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4477.115706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4477.680353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4478.329750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4478.950786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4479.592799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4480.258210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4480.955036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4481.617370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4482.284297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4483.021538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4483.795618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4484.516069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4485.357258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4486.038459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4486.727726] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4486.731570] Lustre: Skipped 41 previous similar messages [ 4486.743509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4486.751988] Lustre: Skipped 41 previous similar messages [ 4486.758313] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4486.762520] Lustre: Skipped 41 previous similar messages [ 4486.767281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4486.777994] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.132@tcp (at 0@lo) [ 4486.780995] Lustre: Skipped 41 previous similar messages [ 4487.415392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4488.064678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4488.760598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4489.609396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4490.303762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4491.088093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4491.738108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4492.495668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4493.163220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4493.908584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4494.571958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4495.285395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4495.866194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4496.540137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4497.287091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4498.031506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4498.692966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4499.349772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4500.034184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4500.679334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4501.370399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4502.135860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4502.876369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4503.690847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4504.527105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4505.301068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4506.081959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4506.924551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4507.787544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4508.585377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4509.550273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4510.506170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4511.214317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4511.971220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4512.880818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4513.655671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4514.488365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4515.191484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4515.981849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4516.757247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4517.514596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4518.414069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4519.100939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4519.900898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4520.665928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4521.422747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4522.274171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4523.173509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4523.933713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4524.616156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4525.370672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4526.131547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4526.884720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4527.629125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4528.384692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4529.027021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4529.720055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4530.373285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4531.016533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4531.666350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4532.242657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4532.802921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4533.539821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4534.214181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4534.942527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4535.611474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4536.239541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4536.875400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4537.544772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4538.295645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4539.051422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4539.723444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4540.412153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4541.088810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4541.773789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4542.511345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4543.390366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4544.080541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4544.792732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4545.661369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4546.421474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4547.103548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4547.906529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4548.775741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4549.501584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4550.307996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4551.135787] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4551.139777] Lustre: Skipped 86 previous similar messages [ 4551.150737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4551.157336] Lustre: Skipped 86 previous similar messages [ 4551.161958] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4551.165816] Lustre: Skipped 86 previous similar messages [ 4551.169624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4551.178094] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.132@tcp (at 0@lo) [ 4551.185962] Lustre: Skipped 86 previous similar messages [ 4551.923288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4552.782684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4553.734334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4554.595765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4555.401188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4556.149567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4556.917170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4557.553115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4558.206508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4558.999970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4559.678356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4560.419501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4561.087390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4561.880157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4562.559222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4563.225853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4563.887634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4564.659918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4565.513957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4566.202244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4566.889852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4567.650104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4568.344762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4569.066963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4569.735904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4570.449959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4571.081579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4571.770773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4572.435678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4573.215554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4574.008914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4574.787986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4575.660866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4576.468037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4577.147711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4577.890686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4578.598248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4579.446314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4580.216962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4580.905032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4581.529867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4582.088319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4582.760833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4583.419573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4584.015368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4584.686761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4585.297321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4585.947530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4586.561481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4587.108545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4587.719303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4588.311995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4588.906344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4589.543769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4590.214281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4590.880268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4591.551041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4592.160912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4592.853832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4593.433408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4594.052252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4594.710666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4595.394366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4596.060481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4596.626968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4597.216483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4597.817534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4598.496707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4599.119861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4599.843900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4600.570799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4601.225329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4601.888732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4602.619730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4603.229773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4603.838767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4604.432388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4605.091842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4605.730121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4606.344664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4607.050758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4607.760537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4608.519536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4609.271776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4610.019061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4610.736638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4611.474244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4612.218366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4612.928638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4613.563683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4614.111583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4614.662912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4615.261294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4615.776880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4616.337303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4616.851530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4617.361272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4617.984518] LustreError: 161864:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4617.990427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4617.991627] LustreError: 161864:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4618.653896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4619.164757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4619.706953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4620.285859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4620.919684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4621.493348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4621.995732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4622.570293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4623.164425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4650.983494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4650.990738] LustreError: Skipped 6 previous similar messages [ 4650.991182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4652.008612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4652.012939] Lustre: Skipped 1 previous similar message [ 4652.485641] LustreError: 178779:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4652.591293] Lustre: server umount lustre-OST0000 complete [ 4655.013553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4655.016345] Lustre: Skipped 1 previous similar message [ 4660.195648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4660.200497] Lustre: Skipped 1 previous similar message [ 4661.088862] LustreError: 178980:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4661.092239] LustreError: 178980:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4661.175362] Lustre: server umount lustre-MDT0000 complete [ 4663.736380] LustreError: 161825:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753746079 with bad export cookie 1524153114298274859 [ 4663.741527] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4663.741659] LustreError: 161825:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4663.843862] LustreError: 179181:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4663.847034] LustreError: 179181:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4663.970243] Lustre: server umount lustre-MDT0001 complete [ 4672.282791] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4674.467586] Key type lgssc unregistered [ 4674.691778] LNet: 179709:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4675.764091] LNet: Removed LNI 192.168.204.132@tcp [ 4676.391364] Key type .llcrypt unregistered [ 4676.397528] Key type ._llcrypt unregistered [ 4692.426203] Key type ._llcrypt registered [ 4692.427622] Key type .llcrypt registered [ 4692.512583] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4693.186901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4693.198701] alg: No test for adler32 (adler32-zlib) [ 4694.234392] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 4694.454884] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4694.458483] LNet: Accept secure, port 988 [ 4696.104273] Key type lgssc registered [ 4696.892946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4702.282927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4702.311485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4703.707862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4706.438328] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4711.991595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4712.226348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4714.789866] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4718.514958] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4720.998138] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4725.665589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4725.908817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4727.999360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3924 to 0x280000401:3969) [ 4730.549125] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4734.641882] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4742.962884] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4743.995622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4744.007946] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4744.013881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4744.023265] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.132@tcp (at 0@lo) [ 4744.024090] LustreError: 181093:0:(osp_precreate.c:1266:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 4747.503649] Lustre: lustre-OST0000: Not available for connect from 192.168.204.32@tcp (stopping) [ 4747.748347] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4747.749714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4751.221148] LustreError: 183449:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4751.315590] Lustre: server umount lustre-OST0000 complete [ 4764.024408] LustreError: 181093:0:(osp_precreate.c:1266:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 4782.497248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4782.762033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4786.921803] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4787.877848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4787.891718] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 4787.892583] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4790.227940] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4791.133712] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4793.031194] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff929590f10800.ost_server_uuid 50 [ 4793.944333] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff929590f10800.ost_server_uuid in FULL state after 0 sec [ 4798.436628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4798.437599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4798.441917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4798.447390] Lustre: Skipped 2 previous similar messages [ 4798.452517] Lustre: Skipped 2 previous similar messages [ 4801.972166] LustreError: 184765:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4801.979835] LustreError: 184765:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4802.112935] Lustre: server umount lustre-OST0000 complete [ 4804.579964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4804.585579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4804.592119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4804.595208] Lustre: Skipped 1 previous similar message [ 4808.673648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4808.674053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4810.446162] LustreError: 184968:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4810.448962] LustreError: 184968:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4810.501947] Lustre: server umount lustre-MDT0000 complete [ 4812.493498] LustreError: 181052:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753746227 with bad export cookie 5633900001877645273 [ 4812.495783] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4812.504485] LustreError: 181052:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4812.602910] LustreError: 185169:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4812.605937] LustreError: 185169:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4812.758679] Lustre: server umount lustre-MDT0001 complete [ 4819.411623] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4821.354251] Key type lgssc unregistered [ 4821.557569] LNet: 185695:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4822.630386] LNet: Removed LNI 192.168.204.132@tcp [ 4823.220083] Key type .llcrypt unregistered [ 4823.222687] Key type ._llcrypt unregistered [ 4842.202022] Key type ._llcrypt registered [ 4842.207478] Key type .llcrypt registered [ 4842.261917] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4842.800271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4842.828440] alg: No test for adler32 (adler32-zlib) [ 4843.763368] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 4843.904596] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4843.907413] LNet: Accept secure, port 988 [ 4845.529476] Key type lgssc registered [ 4846.195721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4851.473287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4851.491220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4852.782420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4854.936341] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4859.430679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4859.653974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4861.910972] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4863.663660] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4867.447952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4867.679890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4871.224557] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4876.639645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4876.687994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4876.792610] Lustre: lustre-OST0001: new disk, initializing [ 4876.795852] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4876.856657] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4877.878794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3971 to 0x280000401:4001) [ 4880.384192] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4881.521528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4881.534559] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4881.574637] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4887.736368] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4893.683842] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4911.073985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4911.078465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4911.087624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4911.758494] LustreError: 190702:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4911.821350] Lustre: server umount lustre-MDT0000 complete [ 4912.610432] LustreError: 187370:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4912.610925] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4912.616910] LustreError: 187370:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4912.622445] Lustre: Skipped 2 previous similar messages [ 4914.048974] LustreError: 187350:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753746329 with bad export cookie 15591296136478202774 [ 4914.050354] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4914.063783] LustreError: 187350:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4916.173155] LustreError: 190903:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4916.182994] LustreError: 190903:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4916.291134] Lustre: server umount lustre-MDT0001 complete [ 4928.574349] LustreError: 191104:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4928.582991] LustreError: 191104:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4928.618488] Lustre: server umount lustre-OST0000 complete [ 4939.834680] LustreError: 191305:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4939.841889] LustreError: 191305:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4939.959445] Lustre: server umount lustre-OST0001 complete [ 4941.635612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4941.682757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4942.172498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4942.227235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4942.710256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4942.760388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4943.256587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4943.286708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4949.531288] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4953.992124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4954.239979] LustreError: 192703:0:(ldlm_lib.c:1113:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4954.279077] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4956.086395] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4959.717363] LustreError: 192703:0:(ldlm_lib.c:1113:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4959.922275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4961.898877] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4963.202960] Lustre: 193801:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4966.259156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4966.418172] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4966.421271] Lustre: Skipped 1 previous similar message [ 4969.031910] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4970.541984] LustreError: 194157:0:(ldlm_lib.c:1113:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4972.941461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4975.536808] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4978.795849] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4004 to 0x280000401:4033) [ 4984.926589] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4986.487545] Lustre: 195635:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4999.652741] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4999.655364] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4999.658982] Lustre: Skipped 2 previous similar messages [ 4999.664647] Lustre: Skipped 3 previous similar messages [ 5002.966977] LustreError: 195943:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5002.970639] LustreError: 195943:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5004.769168] LustreError: 192702:0:(ldlm_lib.c:1113:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5004.775884] LustreError: 192702:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5005.074469] Lustre: server umount mylustre-MDT0000 complete [ 5006.911370] LustreError: 192679:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753746422 with bad export cookie 15591296136478205581 [ 5006.912080] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5006.918143] LustreError: 192679:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5007.137896] Lustre: server umount mylustre-MDT0001 complete [ 5019.133084] LustreError: 196345:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5019.137330] LustreError: 196345:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5019.161791] Lustre: server umount mylustre-OST0000 complete [ 5031.414842] Lustre: server umount mylustre-OST0001 complete [ 5033.138570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5033.195405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5033.762529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5033.821489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5034.376199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5034.427540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5034.954042] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5034.997947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5041.589258] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5046.515520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.791138] LustreError: 197945:0:(ldlm_lib.c:1113:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5046.852881] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5046.855780] Lustre: Skipped 1 previous similar message [ 5048.805365] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5052.723841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5054.528564] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5055.747391] Lustre: 199043:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5058.871697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5062.167188] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5063.211761] LustreError: 199401:0:(ldlm_lib.c:1113:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5063.221953] LustreError: 199401:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5066.462024] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5069.032111] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5072.428248] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4004 to 0x280000401:4065) [ 5072.428268] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5078.021436] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5079.717724] Lustre: 200877:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5088.739403] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5088.744130] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5088.749781] Lustre: Skipped 1 previous similar message [ 5088.755228] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 5092.834834] Lustre: tfs-MDT0000-lwp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5092.836878] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 5092.841989] Lustre: Skipped 2 previous similar messages [ 5092.844527] Lustre: Skipped 2 previous similar messages [ 5094.099932] LustreError: 201127:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5094.103896] LustreError: 201127:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5094.147785] Lustre: server umount tfs-MDT0000 complete [ 5095.885268] LustreError: 197920:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753746511 with bad export cookie 15591296136478207247 [ 5095.888060] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5095.891068] LustreError: 197920:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5096.094389] Lustre: server umount tfs-MDT0001 complete [ 5108.246194] Lustre: server umount tfs-OST0000 complete [ 5120.388163] Lustre: server umount tfs-OST0001 complete [ 5121.844940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5121.894049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5122.412726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5122.465586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5122.974856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5123.017437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5123.550032] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5123.586353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5129.683539] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5134.030476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5134.265790] LustreError: 203132:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5134.318222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5134.321064] Lustre: Skipped 3 previous similar messages [ 5136.229607] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5140.565369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5142.642498] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5144.025332] Lustre: 204229:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5147.015960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5149.592846] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5153.377727] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5155.987881] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5159.722279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4004 to 0x280000401:4097) [ 5159.725948] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5165.008576] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5166.666136] Lustre: 206061:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5174.811864] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 19:49:50 (1753746590) [ 5190.628149] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5190.630076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5190.634334] Lustre: Skipped 2 previous similar messages [ 5190.636929] Lustre: Skipped 2 previous similar messages [ 5193.349743] LustreError: 206429:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5193.352126] LustreError: 206429:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5193.391518] Lustre: server umount lustre-MDT0000 complete [ 5194.943937] LustreError: 203108:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753746610 with bad export cookie 15591296136478208913 [ 5194.945462] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5194.948943] LustreError: 203108:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5199.777166] LustreError: 203128:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5199.777607] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5199.784331] LustreError: 203128:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 5199.786714] Lustre: Skipped 4 previous similar messages [ 5201.215099] Lustre: server umount lustre-MDT0001 complete [ 5209.155123] Lustre: server umount lustre-OST0000 complete [ 5217.055617] Lustre: server umount lustre-OST0001 complete [ 5219.638108] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 5223.874331] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5228.576568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5231.327576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5234.078224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5236.853135] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5239.999086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5240.024462] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 5240.024462] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 5240.024462] [ 5240.039543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 5240.149484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5240.173720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5240.215385] Lustre: lustre-MDT0000: new disk, initializing [ 5240.248901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5240.252200] Lustre: Skipped 3 previous similar messages [ 5240.260790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5241.839844] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5247.355242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5247.393821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5247.438217] Lustre: 209201:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5247.476369] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5247.478888] Lustre: Skipped 1 previous similar message [ 5247.526593] Lustre: lustre-MDT0001: new disk, initializing [ 5247.597976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5247.602549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5249.428651] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5254.749449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5254.780338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5254.896924] Lustre: lustre-OST0000: new disk, initializing [ 5254.899430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5256.457609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5256.461594] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5256.507101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5257.403532] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5262.682752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5262.716627] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5262.784587] Lustre: lustre-OST0001: new disk, initializing [ 5262.787858] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5264.290685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5264.295584] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5264.342181] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5265.389828] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5270.847763] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 19:51:26 (1753746686) [ 5275.822417] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 19:51:31 (1753746691) [ 5289.961061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5289.965201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5289.966771] Lustre: Skipped 4 previous similar messages [ 5293.688106] LustreError: 212597:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5293.693169] LustreError: 212597:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5293.724053] Lustre: server umount lustre-OST0000 complete [ 5298.658934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5301.875560] Lustre: server umount lustre-MDT0000 complete [ 5303.637637] LustreError: 209193:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753746718 with bad export cookie 15591296136478210467 [ 5303.641085] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5303.643080] LustreError: 209193:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5309.883969] Lustre: server umount lustre-MDT0001 complete [ 5315.293970] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5326.405559] Lustre: server umount lustre-OST0001 complete [ 5329.231053] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 5336.704442] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5340.959124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5343.820076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5346.378510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5348.945436] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5351.713623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5351.743781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5351.854566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5351.868864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5351.909830] Lustre: lustre-MDT0000: new disk, initializing [ 5351.944421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5353.470486] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5358.114159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5358.148446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5358.189833] Lustre: 216037:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5358.321508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5359.840510] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5363.259153] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5364.662209] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5367.411415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5367.440996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5369.278354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5369.281615] Lustre: Skipped 1 previous similar message [ 5369.283642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5369.324190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5369.921937] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5373.452323] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5379.554756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5379.557829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5379.558368] LustreError: Skipped 1 previous similar message [ 5379.559144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5379.559148] Lustre: Skipped 7 previous similar messages [ 5379.563874] Lustre: Skipped 5 previous similar messages [ 5382.203585] Lustre: server umount lustre-OST0000 complete [ 5384.161683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5397.984270] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5398.061812] Lustre: server umount lustre-MDT0000 complete [ 5399.010349] LustreError: 216047:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5399.017369] LustreError: 216047:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 5399.479397] LustreError: 216812:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753746814 with bad export cookie 15591296136478211937 [ 5399.481956] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5399.483855] LustreError: 216812:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5399.646664] Lustre: server umount lustre-MDT0001 complete [ 5404.585378] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5405.768140] Key type lgssc unregistered [ 5405.911448] LNet: 219394:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5406.950337] LNet: Removed LNI 192.168.204.132@tcp [ 5407.294903] Key type .llcrypt unregistered [ 5407.296089] Key type ._llcrypt unregistered [ 5418.347182] Key type ._llcrypt registered [ 5418.348845] Key type .llcrypt registered [ 5418.394773] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5418.749261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5418.804776] alg: No test for adler32 (adler32-zlib) [ 5419.690887] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 5419.793727] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 5419.796375] LNet: Accept secure, port 988 [ 5421.393067] Key type lgssc registered [ 5421.802891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5424.821416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5424.828922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5426.032084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5427.647968] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5431.235799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5431.405622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5432.996326] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5435.163871] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5436.518915] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5439.234896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5439.375348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5441.573324] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5443.736437] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5444.159568] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 5446.627366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5446.628298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5446.636995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5450.988133] LustreError: 223082:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5451.054608] Lustre: server umount lustre-OST0000 complete [ 5456.865584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5456.865750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5456.866513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5456.870054] Lustre: Skipped 1 previous similar message [ 5458.555918] LustreError: 223287:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5458.557855] LustreError: 223287:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5458.587499] Lustre: server umount lustre-MDT0000 complete [ 5459.969302] LustreError: 220784:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753746875 with bad export cookie 7595365071610908176 [ 5459.972410] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5459.973132] LustreError: 220784:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5460.041095] LustreError: 223489:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5460.043588] LustreError: 223489:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5460.130138] Lustre: server umount lustre-MDT0001 complete [ 5464.050242] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5465.347523] Key type lgssc unregistered [ 5465.485336] LNet: 224015:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5466.534421] LNet: Removed LNI 192.168.204.132@tcp [ 5466.863043] Key type .llcrypt unregistered [ 5466.864803] Key type ._llcrypt unregistered [ 5473.747693] Key type ._llcrypt registered [ 5473.749335] Key type .llcrypt registered [ 5473.792266] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5474.154767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5474.189059] alg: No test for adler32 (adler32-zlib) [ 5475.068108] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 5475.168675] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 5475.171336] LNet: Accept secure, port 988 [ 5476.760198] Key type lgssc registered [ 5477.133687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5480.468466] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5480.475045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5498.016613] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5504.160204] LustreError: 225154:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.132@tcp: failed processing log, type 4: rc = -110 [ 5534.880224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5537.175737] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5539.266841] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5540.047678] LustreError: 225812:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 5540.125160] Lustre: server umount lustre-OST0000 complete [ 5546.491658] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5547.773383] Key type lgssc unregistered [ 5547.932493] LNet: 226644:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5548.966461] LNet: Removed LNI 192.168.204.132@tcp [ 5549.344466] Key type .llcrypt unregistered [ 5549.345670] Key type ._llcrypt unregistered [ 5564.905785] Key type ._llcrypt registered [ 5564.907462] Key type .llcrypt registered [ 5564.956496] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5565.409765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5565.421621] alg: No test for adler32 (adler32-zlib) [ 5566.301295] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 5566.397709] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 5566.400408] LNet: Accept secure, port 988 [ 5567.993388] Key type lgssc registered [ 5568.440818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5573.868554] loop0: detected capacity change from 0 to 204800000 [ 5573.934778] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5573.939853] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5574.094548] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5574.100757] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5574.129841] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5574.154948] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5574.180594] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5574.211988] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5574.249806] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5574.279747] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5574.710232] loop0: detected capacity change from 0 to 204800000 [ 5574.781428] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5575.064214] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5575.450294] loop0: detected capacity change from 0 to 204800000 [ 5575.602020] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5575.998912] loop0: detected capacity change from 0 to 204800000 [ 5576.150356] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5576.501150] loop0: detected capacity change from 0 to 204800000 [ 5576.512694] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5577.965885] loop0: detected capacity change from 0 to 204800000 [ 5577.978694] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5579.549171] loop0: detected capacity change from 0 to 204800000 [ 5579.563477] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5581.129996] loop0: detected capacity change from 0 to 204800000 [ 5581.147549] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5582.748096] loop0: detected capacity change from 0 to 204800000 [ 5582.799877] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5582.828652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5582.837668] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5582.870182] Lustre: 230010:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5582.874290] Lustre: 230010:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 5582.881171] Lustre: 230010:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 5582.886037] Lustre: 230010:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 5582.890384] Lustre: 230010:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 5582.896321] Lustre: 230010:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 5582.902690] Lustre: 230010:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 5582.907898] Lustre: 230010:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 5582.913254] Lustre: 230010:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 5582.919054] Lustre: 230010:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 5582.925355] Lustre: 230010:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 5582.931560] Lustre: 230010:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 5582.937875] Lustre: 230010:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 5582.944421] Lustre: 230010:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 5585.838235] LustreError: 230276:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5585.894953] Lustre: server umount MGS complete [ 5586.259158] loop0: detected capacity change from 0 to 204800000 [ 5586.297491] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5586.485137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5586.506797] Lustre: lustre-MDT0000: Aborting client recovery [ 5586.879330] loop1: detected capacity change from 0 to 204800000 [ 5586.935695] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5586.965583] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5586.993759] Lustre: 230444:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5586.998748] Lustre: 230444:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 5587.004277] Lustre: 230444:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 5587.009278] Lustre: 230444:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 5587.013962] Lustre: 230444:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 5587.019177] Lustre: 230444:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 5587.025163] Lustre: 230444:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 5587.034422] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5587.122591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5587.144131] Lustre: lustre-MDT0001: Aborting client recovery [ 5587.517754] loop2: detected capacity change from 0 to 204800000 [ 5587.556782] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5587.595643] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5587.619139] Lustre: 230529:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5587.624830] Lustre: 230529:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 5587.630110] Lustre: 230529:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 5587.635671] Lustre: 230529:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 5587.640176] Lustre: 230529:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 5587.644981] Lustre: 230529:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 5587.649533] Lustre: 230529:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 5587.703944] Lustre: Found index 0 for lustre-OST0000, updating log [ 5587.757780] Lustre: lustre-OST0000: Aborting recovery [ 5588.145128] loop3: detected capacity change from 0 to 204800000 [ 5588.193717] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5588.252449] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5588.271488] Lustre: 230616:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5588.276604] Lustre: 230616:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 5588.281681] Lustre: 230616:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 5588.286606] Lustre: 230616:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 5588.291365] Lustre: 230616:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 5588.296436] Lustre: 230616:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 5588.301222] Lustre: 230616:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 5588.345469] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5588.348148] Lustre: Skipped 1 previous similar message [ 5588.355691] Lustre: lustre-OST0001: Aborting recovery [ 5593.584715] LustreError: 230387:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -108 [ 5596.645458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 5596.645736] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 5596.664481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 5596.665838] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 5597.671547] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 5597.671709] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 5599.948311] LustreError: 230355:0:(mdt_identity.c:95:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 357us: rc = -2 [ 5613.027107] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5613.027917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5613.028029] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5613.032969] Lustre: Skipped 3 previous similar messages [ 5613.035698] Lustre: Skipped 2 previous similar messages [ 5617.340270] LustreError: 230971:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5617.342907] LustreError: 230971:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5617.375647] Lustre: server umount lustre-MDT0000 complete [ 5617.721688] LustreError: 230338:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747032 with bad export cookie 4020761048769069106 [ 5617.723873] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5617.725779] LustreError: 230338:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5618.144926] LustreError: 230353:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5618.145324] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5618.145707] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5618.153051] LustreError: 230353:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5618.158255] Lustre: Skipped 1 previous similar message [ 5622.176661] LustreError: 230353:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5622.176810] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5622.184753] LustreError: 230353:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5622.190461] Lustre: Skipped 2 previous similar messages [ 5623.868714] LustreError: 231022:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5623.871590] LustreError: 231022:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5623.966798] Lustre: server umount lustre-MDT0001 complete [ 5630.382213] LustreError: 231073:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5630.385037] LustreError: 231073:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5630.420503] Lustre: server umount lustre-OST0000 complete [ 5645.280461] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5645.300642] LustreError: 231124:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5645.303872] LustreError: 231124:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5645.398990] Lustre: server umount lustre-OST0001 complete [ 5649.343330] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 19:57:44 (1753747064) [ 5662.987730] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 5666.854834] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5671.282432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5674.140101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5677.132528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5680.340061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5683.627764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5683.665805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5683.791807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5683.815721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5683.864802] Lustre: lustre-MDT0000: new disk, initializing [ 5683.927090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5683.941605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5685.857418] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5691.516362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5691.551574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5691.601539] Lustre: 234654:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5691.643836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5691.646475] Lustre: Skipped 1 previous similar message [ 5691.690246] Lustre: lustre-MDT0001: new disk, initializing [ 5691.742624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5691.765839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5691.771927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5693.568931] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5697.587917] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5699.383780] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5702.496849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5702.533303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5702.660156] Lustre: lustre-OST0000: new disk, initializing [ 5702.664137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5702.702622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5705.395042] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5706.435338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5706.439903] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5706.483297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5709.169861] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5712.115772] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 5716.451281] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 5724.250453] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 5763.043516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5763.045819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5763.047196] LustreError: Skipped 1 previous similar message [ 5763.053292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5763.056249] Lustre: Skipped 1 previous similar message [ 5768.163440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5768.166354] Lustre: Skipped 1 previous similar message [ 5775.328182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5775.354140] LustreError: 238199:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5775.356634] LustreError: 238199:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5775.457905] Lustre: server umount lustre-OST0000 complete [ 5778.402051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.402755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5778.408023] Lustre: Skipped 2 previous similar messages [ 5778.414166] Lustre: Skipped 2 previous similar messages [ 5783.700699] Lustre: server umount lustre-MDT0000 complete [ 5785.428187] LustreError: 236388:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747200 with bad export cookie 4020761048769071815 [ 5785.430875] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5785.432621] LustreError: 236388:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5785.596129] Lustre: server umount lustre-MDT0001 complete [ 5789.277376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5790.902893] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5792.866640] LustreError: 239539:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5792.869022] LustreError: 239539:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5792.901680] Lustre: server umount MGS complete [ 5797.600255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5797.817812] LustreError: 240162:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5797.830397] LustreError: 240174:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 5797.856269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5799.524320] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5802.977269] LustreError: 240162:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5803.422601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5805.161939] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5807.227674] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5808.619706] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5811.064472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5813.286594] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5815.306405] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5849.057372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5849.061123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5849.068074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5849.070805] Lustre: Skipped 3 previous similar messages [ 5854.005216] LustreError: 242831:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5854.008017] LustreError: 242831:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5854.097734] Lustre: server umount lustre-OST0000 complete [ 5859.809925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5859.813955] LustreError: Skipped 1 previous similar message [ 5859.815546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5859.820551] Lustre: Skipped 1 previous similar message [ 5861.884906] Lustre: server umount lustre-MDT0000 complete [ 5863.611343] LustreError: 242557:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747278 with bad export cookie 4020761048769073642 [ 5863.613487] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5863.616675] LustreError: 242557:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5863.787665] Lustre: server umount lustre-MDT0001 complete [ 5868.516550] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5869.914087] Key type lgssc unregistered [ 5870.068451] LNet: 243759:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5871.081757] LNet: Removed LNI 192.168.204.132@tcp [ 5871.477083] Key type .llcrypt unregistered [ 5871.478902] Key type ._llcrypt unregistered [ 5887.961770] Key type ._llcrypt registered [ 5887.963313] Key type .llcrypt registered [ 5888.030259] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 5895.388623] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5895.909184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5895.921278] alg: No test for adler32 (adler32-zlib) [ 5896.799734] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 5896.917512] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 5896.920133] LNet: Accept secure, port 988 [ 5898.520176] Key type lgssc registered [ 5898.983954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5903.161478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5905.993980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5908.489167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5911.335837] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5914.372220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5914.400935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5914.410937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5915.517034] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5915.534142] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5915.573374] Lustre: lustre-MDT0000: new disk, initializing [ 5915.611280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5915.620529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5917.188753] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5922.453318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5922.485443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5922.540193] Lustre: 247647:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5922.576132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5922.579278] Lustre: Skipped 1 previous similar message [ 5922.623380] Lustre: lustre-MDT0001: new disk, initializing [ 5922.675496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5922.695387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5922.699951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5924.373103] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5928.274580] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5929.983537] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5933.333163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5933.383787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5933.528559] Lustre: lustre-OST0000: new disk, initializing [ 5933.532906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5933.582371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5933.616935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5933.623790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5933.696327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5936.153942] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5940.225878] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5944.240024] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 5952.891682] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 5957.909619] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 5962.504434] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 6010.337814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6010.341737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6010.347709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6010.852843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6010.856769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6010.865638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6012.403121] LustreError: 251242:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6012.487911] Lustre: server umount lustre-OST0000 complete [ 6014.947488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6014.951224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6014.957576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6020.424318] LustreError: 251443:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6020.427784] LustreError: 251443:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6020.479283] Lustre: server umount lustre-MDT0000 complete [ 6022.199333] LustreError: 248421:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747437 with bad export cookie 10531769758982267580 [ 6022.203146] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6022.204305] LustreError: 248421:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6022.284328] LustreError: 251645:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6022.287044] LustreError: 251645:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6022.380172] Lustre: server umount lustre-MDT0001 complete [ 6025.978855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6027.584284] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6029.610716] LustreError: 252580:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6029.613090] LustreError: 252580:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6029.645382] Lustre: server umount MGS complete [ 6034.186348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6034.382738] LustreError: 253202:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6034.391682] LustreError: 253202:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6034.398443] LustreError: 253214:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 6034.428791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6035.950647] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6039.522079] LustreError: 253202:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6039.655875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6039.794495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6041.301911] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6043.290522] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6044.609712] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6047.155981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6049.386650] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6051.519269] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6079.973298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6079.974976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6079.977187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6079.977192] Lustre: Skipped 2 previous similar messages [ 6079.987195] Lustre: Skipped 2 previous similar messages [ 6085.091082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6085.095265] Lustre: Skipped 2 previous similar messages [ 6085.810433] LustreError: 255866:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6085.812581] LustreError: 255866:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6085.894260] Lustre: server umount lustre-OST0000 complete [ 6090.210393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6090.215812] Lustre: Skipped 1 previous similar message [ 6093.584584] Lustre: server umount lustre-MDT0000 complete [ 6094.935047] LustreError: 253181:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747510 with bad export cookie 10531769758982269435 [ 6094.938151] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6094.939917] LustreError: 253181:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6094.999884] LustreError: 256269:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6095.002020] LustreError: 256269:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6095.081631] Lustre: server umount lustre-MDT0001 complete [ 6099.888529] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6101.066244] Key type lgssc unregistered [ 6101.216107] LNet: 256795:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6102.247424] LNet: Removed LNI 192.168.204.132@tcp [ 6102.602590] Key type .llcrypt unregistered [ 6102.604803] Key type ._llcrypt unregistered [ 6110.319602] Key type ._llcrypt registered [ 6110.321211] Key type .llcrypt registered [ 6110.372208] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 6116.809931] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6117.168390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6117.209041] alg: No test for adler32 (adler32-zlib) [ 6118.089098] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 6118.185811] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 6118.188265] LNet: Accept secure, port 988 [ 6119.776188] Key type lgssc registered [ 6120.199302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6124.015116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6126.509636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6129.866557] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6132.974532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6135.585798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6135.610485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6135.619258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6136.721831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6136.737496] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6136.778554] Lustre: lustre-MDT0000: new disk, initializing [ 6136.810911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6136.819969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6138.371122] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6143.016063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6143.047917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6143.176957] Lustre: lustre-OST0000: new disk, initializing [ 6143.179650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6143.182183] Lustre: Skipped 1 previous similar message [ 6143.210381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6145.342195] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6152.714669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6152.720384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6152.736867] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6403.552889] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6403.558657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6408.673097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6409.282060] LustreError: 261678:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6409.481727] Lustre: server umount lustre-MDT0000 complete [ 6410.807163] LustreError: 259930:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747826 with bad export cookie 6887298255343406521 [ 6410.808830] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6410.811297] LustreError: 259930:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6410.840254] LustreError: 261883:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6410.843063] LustreError: 261883:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6410.910210] Lustre: server umount lustre-OST0000 complete [ 6419.754886] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 6423.557492] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6427.581670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6430.121211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6432.322869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6434.713242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6437.248749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6437.271540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6437.364827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6437.379472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6437.415928] Lustre: lustre-MDT0000: new disk, initializing [ 6437.442819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6437.450649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6438.868878] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6443.378376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6443.404778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6443.444254] Lustre: 264903:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6443.469787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6443.472279] Lustre: Skipped 1 previous similar message [ 6443.508176] Lustre: lustre-MDT0001: new disk, initializing [ 6443.544218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6443.556146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6443.560465] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6444.925671] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6448.184821] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6449.532397] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6452.307065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6452.338515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6452.443501] Lustre: lustre-OST0000: new disk, initializing [ 6452.446268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6452.478184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6454.078123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6454.082748] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6454.123883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6454.549238] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6457.780522] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6459.135216] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6459.215178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6460.717340] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6460.801798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6464.482669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6464.483352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6464.486208] LustreError: Skipped 1 previous similar message [ 6464.486878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6464.491741] Lustre: Skipped 1 previous similar message [ 6467.567108] LustreError: 267808:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6467.570401] LustreError: 267808:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6467.640705] Lustre: server umount lustre-OST0000 complete [ 6469.089069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6469.092284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6469.097711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6469.099818] Lustre: Skipped 1 previous similar message [ 6474.721172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6474.723163] Lustre: Skipped 2 previous similar messages [ 6475.137087] LustreError: 268009:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6475.140033] LustreError: 268009:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6475.169239] Lustre: server umount lustre-MDT0000 complete [ 6476.535032] LustreError: 268010:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747891 with bad export cookie 6887298255346347312 [ 6476.538152] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6476.539537] LustreError: 268010:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6476.698791] Lustre: server umount lustre-MDT0001 complete [ 6478.031915] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 6478.680913] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 20:11:34 (1753747894) [ 6480.990762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6481.156439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6482.577322] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6485.635134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6487.172383] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6489.282528] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6490.694912] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6493.275892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6495.536833] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6497.516107] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6500.103167] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6500.140215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6500.208898] Lustre: lustre-OST0001: new disk, initializing [ 6500.212602] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6500.254370] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6500.257766] Lustre: Skipped 2 previous similar messages [ 6502.466302] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6506.494499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6506.498199] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6506.514497] LustreError: 271814:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 6506.519105] LustreError: 271814:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 6507.555411] LustreError: 271814:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 6507.559183] LustreError: 271814:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 6508.579271] LustreError: 271819:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 6508.583397] LustreError: 271819:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 6510.626994] LustreError: 271814:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 6510.631547] LustreError: 271814:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 6511.715374] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6511.792944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6513.006264] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9295a0de2000.ost_server_uuid 50 [ 6513.517577] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9295a0de2000.ost_server_uuid in FULL state after 0 sec [ 6514.724832] LustreError: 271814:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 6514.728410] LustreError: 271814:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 6514.817879] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9295a0de2000.ost_server_uuid 50 [ 6515.372381] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9295a0de2000.ost_server_uuid in FULL state after 0 sec [ 6516.782635] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6530.790624] Lustre: lustre-OST0001: Not available for connect from 192.168.204.32@tcp (stopping) [ 6532.074082] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6532.074177] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6532.079336] Lustre: Skipped 1 previous similar message [ 6532.085021] LustreError: Skipped 1 previous similar message [ 6533.485736] LustreError: 272558:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6533.487688] LustreError: 272558:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6533.507167] Lustre: server umount lustre-OST0001 complete [ 6537.187617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6537.189896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6537.191438] LustreError: 272450:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6537.194503] Lustre: Skipped 2 previous similar messages [ 6537.200250] LustreError: 272450:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6541.356200] Lustre: server umount lustre-OST0000 complete [ 6547.426521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6547.426575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6547.427567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6547.427572] Lustre: Skipped 4 previous similar messages [ 6547.440436] Lustre: Skipped 1 previous similar message [ 6557.152193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6557.199818] LustreError: 272961:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6557.202827] LustreError: 272961:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6557.239408] Lustre: server umount lustre-MDT0000 complete [ 6557.665247] LustreError: 268785:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6557.671352] LustreError: 268785:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6558.721424] LustreError: 268765:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753747973 with bad export cookie 6887298255346348180 [ 6558.723805] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6558.726428] LustreError: 268765:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6558.885739] Lustre: server umount lustre-MDT0001 complete [ 6563.932347] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6565.097234] Key type lgssc unregistered [ 6565.238630] LNet: 273689:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6566.310303] LNet: Removed LNI 192.168.204.132@tcp [ 6566.637926] Key type .llcrypt unregistered [ 6566.639784] Key type ._llcrypt unregistered [ 6576.197294] Key type ._llcrypt registered [ 6576.198507] Key type .llcrypt registered [ 6576.243484] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6576.717837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6576.733911] alg: No test for adler32 (adler32-zlib) [ 6577.612523] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 6577.710419] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 6577.712912] LNet: Accept secure, port 988 [ 6579.304253] Key type lgssc registered [ 6579.691563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6582.378986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6582.386974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6583.555389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6584.854130] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6586.730890] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6589.061449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6589.195866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6590.571681] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6594.097727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6594.238718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6596.406987] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6598.451310] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6599.397490] LustreError: 276595:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6599.405717] LustreError: 276595:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6600.990152] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6601.086642] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6603.213642] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6605.302992] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6607.978154] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6607.978291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 6612.189736] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6612.290126] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6695.399369] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6695.400628] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6695.404551] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6695.407789] LustreError: Skipped 1 previous similar message [ 6695.415708] Lustre: Skipped 1 previous similar message [ 6698.613183] LustreError: 278440:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6698.646810] Lustre: server umount lustre-OST0001 complete [ 6700.519385] LustreError: 278027:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6700.525278] LustreError: 278027:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6702.050983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6702.054123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6702.059397] Lustre: Skipped 1 previous similar message [ 6702.062537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6705.635312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6705.638794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6705.641033] LustreError: 276594:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6705.641216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6705.654406] LustreError: 276594:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6710.754516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6710.754835] LustreError: 277253:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6710.757016] Lustre: Skipped 2 previous similar messages [ 6710.762834] LustreError: 277253:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6715.360330] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6715.386084] LustreError: 278642:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6715.389101] LustreError: 278642:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6715.472325] Lustre: server umount lustre-OST0000 complete [ 6717.409255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6717.412737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6717.420900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6726.113091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6726.115231] Lustre: Skipped 3 previous similar messages [ 6731.232155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6731.273763] LustreError: 278846:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6731.276361] LustreError: 278846:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6731.308147] Lustre: server umount lustre-MDT0000 complete [ 6732.741600] LustreError: 275036:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748148 with bad export cookie 3543846941585016903 [ 6732.746369] LustreError: 275036:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6732.746419] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6732.923045] Lustre: server umount lustre-MDT0001 complete [ 6737.961566] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6739.175146] Key type lgssc unregistered [ 6739.318531] LNet: 279576:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6740.390427] LNet: Removed LNI 192.168.204.132@tcp [ 6740.766629] Key type .llcrypt unregistered [ 6740.768115] Key type ._llcrypt unregistered [ 6761.029070] Key type ._llcrypt registered [ 6761.030474] Key type .llcrypt registered [ 6761.071431] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6761.426323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6761.472086] alg: No test for adler32 (adler32-zlib) [ 6762.346142] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 6762.453486] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 6762.456446] LNet: Accept secure, port 988 [ 6764.056191] Key type lgssc registered [ 6764.466385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6767.850255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6767.859522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6769.046962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6770.437652] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6773.619870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6773.788325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6775.292591] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6776.389942] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6778.944435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6779.084522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6780.135717] LustreError: 283428:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6780.139162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 6780.142645] LustreError: 283428:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6781.214135] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6784.229963] LustreError: 283428:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6784.237551] LustreError: 283428:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6784.560360] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6784.650305] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6786.711690] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6790.132171] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 6791.077232] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6795.039723] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6804.449317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6804.453382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6804.460469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6805.474454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6805.480378] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6805.484660] Lustre: Skipped 2 previous similar messages [ 6810.128488] LustreError: 285200:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6810.168441] Lustre: server umount lustre-MDT0000 complete [ 6810.596112] LustreError: 284829:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6810.602534] LustreError: 284829:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6811.665492] LustreError: 281793:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748226 with bad export cookie 3912338072493270399 [ 6811.669251] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6811.670496] LustreError: 281793:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6811.739385] LustreError: 285403:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6811.741400] LustreError: 285403:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6811.833727] Lustre: server umount lustre-MDT0001 complete [ 6822.901119] LustreError: 285603:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6822.903208] LustreError: 285603:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6822.919794] Lustre: server umount lustre-OST0000 complete [ 6834.550025] LustreError: 285804:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6834.552729] LustreError: 285804:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6834.637068] Lustre: server umount lustre-OST0001 complete [ 6840.584913] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6844.549620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6844.765764] LustreError: 286970:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6844.804699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6846.446419] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6849.626633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6851.115723] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6852.118903] Lustre: 288066:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6854.400351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6854.522414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6854.524184] Lustre: Skipped 1 previous similar message [ 6855.523208] LustreError: 288422:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6855.527097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 6856.439791] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6859.279384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6860.401803] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 6861.327449] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6864.836887] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6866.109358] Lustre: 289898:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6875.617045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6875.620062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6875.625558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6875.628175] Lustre: Skipped 3 previous similar messages [ 6878.285141] LustreError: 290003:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6878.288331] LustreError: 290003:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6878.335610] Lustre: server umount lustre-MDT0000 complete [ 6879.827382] LustreError: 289901:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748295 with bad export cookie 3912338072493272086 [ 6879.831445] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6879.833295] LustreError: 289901:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6879.993585] Lustre: server umount lustre-MDT0001 complete [ 6891.765780] LustreError: 290407:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6891.768812] LustreError: 290407:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6891.788608] Lustre: server umount lustre-OST0000 complete [ 6903.439484] Lustre: server umount lustre-OST0001 complete [ 6904.899298] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 6905.484364] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 6906.105214] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 20:18:41 (1753748321) [ 6913.940340] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6917.175248] loop0: detected capacity change from 0 to 1099511627776 [ 6917.337779] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 6917.340873] *** VALIDATE xfs *** [ 6917.345140] XFS (loop0): Mounting V5 Filesystem [ 6917.353571] XFS (loop0): Ending clean mount [ 6917.779603] loop1: detected capacity change from 0 to 18691697672192 [ 6933.292313] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6939.510125] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 6942.846944] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6946.457049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6948.927612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6951.399753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6953.919127] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6956.476487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6956.507642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6956.605832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6956.622140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6956.664218] Lustre: lustre-MDT0000: new disk, initializing [ 6956.696594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6956.699268] Lustre: Skipped 1 previous similar message [ 6956.705724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6958.187983] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6962.805797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6962.833856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6962.877438] Lustre: 294951:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6962.914952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6962.916890] Lustre: Skipped 1 previous similar message [ 6962.956980] Lustre: lustre-MDT0001: new disk, initializing [ 6963.020332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6963.025833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6964.449502] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6967.775298] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6969.142630] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6971.695453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6971.724804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6971.831544] Lustre: lustre-OST0000: new disk, initializing [ 6971.833724] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6973.562051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6973.565947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6973.602547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6973.887482] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6977.084141] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6978.495935] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6978.580872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6979.960000] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6980.044834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6984.164094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6984.165286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6984.166646] LustreError: Skipped 1 previous similar message [ 6984.169460] Lustre: Skipped 3 previous similar messages [ 6984.170706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6984.174364] Lustre: Skipped 5 previous similar messages [ 6986.797481] LustreError: 297858:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6986.800122] LustreError: 297858:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6986.862772] Lustre: server umount lustre-OST0000 complete [ 6988.768917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6988.771181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.775491] Lustre: Skipped 1 previous similar message [ 6988.777215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6994.261166] Lustre: server umount lustre-MDT0000 complete [ 6994.401817] LustreError: 294962:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6994.406055] LustreError: 294962:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6995.430289] LustreError: 294943:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748410 with bad export cookie 3912338072493273640 [ 6995.432819] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6995.434691] LustreError: 294943:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6995.579274] Lustre: server umount lustre-MDT0001 complete [ 6998.264372] XFS (loop0): Unmounting Filesystem [ 6999.933969] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 20:20:15 (1753748415) [ 7002.383469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7002.582573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7002.584975] Lustre: Skipped 2 previous similar messages [ 7004.038804] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7007.131627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7008.606275] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7010.669753] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7011.994866] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7014.608347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7016.876978] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7019.176240] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7033.316733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7033.317110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7033.318166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7033.318174] Lustre: Skipped 2 previous similar messages [ 7033.322062] Lustre: Skipped 2 previous similar messages [ 7039.212117] LustreError: 301425:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7039.215305] LustreError: 301425:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7039.291763] Lustre: server umount lustre-OST0000 complete [ 7043.040930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7046.805467] Lustre: server umount lustre-MDT0000 complete [ 7048.099393] LustreError: 299058:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753748463 with bad export cookie 3912338072493274522 [ 7048.102458] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7048.103508] LustreError: 299058:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7048.241121] Lustre: server umount lustre-MDT0001 complete [ 7053.252244] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 7054.344767] Key type lgssc unregistered [ 7054.480417] LNet: 302354:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7055.526341] LNet: Removed LNI 192.168.204.132@tcp [ 7055.837955] Key type .llcrypt unregistered [ 7055.839650] Key type ._llcrypt unregistered [ 7065.976071] Key type ._llcrypt registered [ 7065.977395] Key type .llcrypt registered [ 7066.020280] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 7066.373978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7066.441180] alg: No test for adler32 (adler32-zlib) [ 7067.301648] Lustre: Lustre: Build Version: 2.16.56_105_g447ed16 [ 7067.388624] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 7067.390809] LNet: Accept secure, port 988 [ 7068.976205] Key type lgssc registered [ 7069.338187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7072.007316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7072.015590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7073.176246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7074.556335] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7077.722228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7077.873664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7079.300453] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7081.211207] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7082.532601] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7085.112818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7085.251436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7087.314352] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7089.283229] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7103.456899] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7103.459440] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7103.464509] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7103.971394] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7103.975430] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7108.346753] LustreError: 306167:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7108.368935] Lustre: server umount lustre-MDT0001 complete [ 7109.090465] LustreError: 304585:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7109.096921] LustreError: 304585:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 7109.333031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7109.854107] LustreError: 305139:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7111.976150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7112.026289] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7112.029219] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7112.045235] Lustre: 303857:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7112.158327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7113.477143] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7117.289604] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 7118.310302] LustreError: 307241:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7118.312956] LustreError: 307241:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7118.316276] LustreError: 303860:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7118.333828] LustreError: 307244:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.204.132@tcp: cfg command failed: rc = -17 [ 7118.337974] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 7118.337974] [ 7417.061685] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.32@tcp (stopping) [ 7417.064666] Lustre: Skipped 1 previous similar message [ 7417.315803] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7422.173204] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.32@tcp (stopping) [ 7422.176244] Lustre: Skipped 2 previous similar messages [ 7422.204887] LustreError: 318630:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7422.207646] LustreError: 318630:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7422.232968] Lustre: server umount lustre-MDT0001 complete [ 7422.434159] LustreError: 303869:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7422.440631] LustreError: 303869:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 7423.249609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7425.780675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7425.810652] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7425.812799] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7425.827072] Lustre: 303858:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7425.951720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7427.200913] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7431.141252] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 7434.727559] LustreError: 319815:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7434.730242] LustreError: 319815:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7434.733552] LustreError: 303860:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7434.736373] LustreError: 303860:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 7434.756101] LustreError: 319824:0:(obd_config.c:685:class_setup()) Device 8 already setup (type osp) [ 7434.758162] LustreError: 319824:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.204.132@tcp: cfg command failed: rc = -17 [ 7434.760582] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.132@tcp [ 7434.760582] [ 7729.379788] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.32@tcp (stopping) [ 7733.218810] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7733.219617] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7733.224794] Lustre: Skipped 1 previous similar message [ 7735.167645] LustreError: 331194:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7735.169800] LustreError: 331194:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7735.201233] Lustre: server umount lustre-MDT0001 complete [ 7736.225903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7738.338625] LustreError: 305143:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7738.342966] LustreError: 305143:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 7738.706927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7738.738899] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7738.741511] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7738.756273] Lustre: 303857:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7738.865342] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7740.066479] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7743.978288] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 7744.485433] LustreError: 332263:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7744.489060] LustreError: 332263:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7744.491320] LustreError: 303860:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7744.493153] LustreError: 303860:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 7744.508791] LustreError: 332266:0:(lod_lov.c:207:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 7744.513622] LustreError: 332266:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.204.132@tcp: cfg command failed: rc = -114 [ 7744.516420] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 7744.516420] [ 8056.290158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8056.291758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8056.296079] Lustre: Skipped 1 previous similar message [ 8056.298232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8056.300328] Lustre: Skipped 5 previous similar messages [ 8059.772559] LustreError: 343921:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8059.774424] LustreError: 343921:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8059.798975] Lustre: server umount lustre-MDT0000 complete [ 8061.032124] LustreError: 303849:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749476 with bad export cookie 6133116254819864495 [ 8061.038049] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8061.409232] LustreError: 303863:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8067.132760] LustreError: 344122:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8067.134732] LustreError: 344122:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8067.208088] Lustre: server umount lustre-MDT0001 complete [ 8068.521194] Lustre: server umount lustre-OST0000 complete [ 8071.355103] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 8074.363667] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 8077.887528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8080.210417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8082.335923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8084.488419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8086.631492] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 20:38:22 (1753749502) [ 8088.878040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8088.900581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8088.977521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8088.988303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8089.026770] Lustre: lustre-MDT0000: new disk, initializing [ 8089.052919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8089.059857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8090.366733] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8094.530058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8094.559251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8094.598775] Lustre: 346837:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8094.642840] LustreError: 346847:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8094.645764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8094.646529] LustreError: 346847:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 8094.648244] Lustre: Skipped 1 previous similar message [ 8094.684613] Lustre: lustre-MDT0001: new disk, initializing [ 8094.719457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8094.734063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8094.738260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8096.116960] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8099.272438] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8100.527980] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8102.858849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8102.889546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8102.990531] Lustre: lustre-OST0000: new disk, initializing [ 8102.992681] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8103.025444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8104.865122] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8108.011859] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8109.045821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8109.048697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8109.078598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8119.268169] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8119.270053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8119.271576] Lustre: Skipped 5 previous similar messages [ 8119.274822] Lustre: Skipped 4 previous similar messages [ 8121.979594] LustreError: 349264:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8121.981485] LustreError: 349264:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 8122.010690] Lustre: server umount lustre-MDT0000 complete [ 8123.217313] LustreError: 346827:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753749538 with bad export cookie 6133116254819866448 [ 8123.222523] LustreError: 346827:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8123.222744] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8123.226245] LustreError: 347628:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 8123.390626] Lustre: server umount lustre-MDT0001 complete [ 8133.891820] Lustre: server umount lustre-OST0000 complete [ 8143.305723] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 8146.606798] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 8150.735373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8153.471456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8155.479559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8157.879287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8158.459889] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7898 sec ========== 20:39:33 (1753749573) [ 8159.059825] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:39:34 (1753749574) === [ 8160.291569] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:39:35 (1753749575) === [ 8174.714673] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 8175.811411] Key type lgssc unregistered [ 8175.952361] LNet: 354745:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8176.998298] LNet: Removed LNI 192.168.204.132@tcp [ 8177.299415] Key type .llcrypt unregistered [ 8177.301270] Key type ._llcrypt unregistered