[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1094815243 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002341] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009122] LSM: Security Framework initializing [ 0.010042] Yama: becoming mindful. [ 0.011029] SELinux: Initializing. [ 0.012000] *** VALIDATE selinux *** [ 0.019575] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023756] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026061] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027098] *** VALIDATE tmpfs *** [ 0.029221] *** VALIDATE proc *** [ 0.030215] *** VALIDATE cgroup *** [ 0.031006] *** VALIDATE cgroup2 *** [ 0.032234] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034122] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036026] Spectre V2 : User space: Vulnerable [ 0.037007] Speculative Store Bypass: Vulnerable [ 0.040324] debug: unmapping init [mem 0xffffffffaf259000-0xffffffffaf260fff] [ 0.042390] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043759] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044025] ... version: 2 [ 0.045010] ... bit width: 48 [ 0.046009] ... generic registers: 4 [ 0.047012] ... value mask: 0000ffffffffffff [ 0.048012] ... max period: 00007fffffffffff [ 0.049012] ... fixed-purpose events: 3 [ 0.050009] ... event mask: 000000070000000f [ 0.051267] rcu: Hierarchical SRCU implementation. [ 0.053546] smp: Bringing up secondary CPUs ... [ 0.054512] x86: Booting SMP configuration: [ 0.055018] .... node #0, CPUs: #1 #2 #3 [ 0.063734] smp: Brought up 1 node, 4 CPUs [ 0.066014] smpboot: Max logical packages: 1 [ 0.067012] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.114292] node 0 deferred pages initialised in 45ms [ 0.123089] devtmpfs: initialized [ 0.124244] x86/mm: Memory block size: 128MB [ 0.128399] gcov: version magic: 0x41383552 [ 0.130991] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.132639] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.133282] pinctrl core: initialized pinctrl subsystem [ 0.134217] [ 0.134683] ************************************************************* [ 0.135019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.136020] ** ** [ 0.137022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.138159] ** ** [ 0.139014] ** This means that this kernel is built to expose internal ** [ 0.140014] ** IOMMU data structures, which may compromise security on ** [ 0.141012] ** your system. ** [ 0.142010] ** ** [ 0.143240] ** If you see this message and you are not debugging the ** [ 0.144012] ** kernel, report this immediately to your vendor! ** [ 0.145014] ** ** [ 0.146013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.147019] ************************************************************* [ 0.148808] NET: Registered protocol family 16 [ 0.149692] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.150055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.151391] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.152785] cpuidle: using governor menu [ 0.153957] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.156541] PCI: Using configuration type 1 for base access [ 0.159127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.168499] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.173082] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.175045] cryptd: max_cpu_qlen set to 1000 [ 0.178414] ACPI: Added _OSI(Module Device) [ 0.179008] ACPI: Added _OSI(Processor Device) [ 0.180010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.181009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.185860] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.189136] ACPI: Interpreter enabled [ 0.190049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.191010] ACPI: Using IOAPIC for interrupt routing [ 0.192087] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.193388] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.204551] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.205134] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.206020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.207096] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.210126] acpiphp: Slot [2] registered [ 0.211145] acpiphp: Slot [5] registered [ 0.212192] acpiphp: Slot [6] registered [ 0.213231] acpiphp: Slot [7] registered [ 0.214275] acpiphp: Slot [8] registered [ 0.215117] acpiphp: Slot [9] registered [ 0.216151] acpiphp: Slot [10] registered [ 0.217119] acpiphp: Slot [3] registered [ 0.218131] acpiphp: Slot [4] registered [ 0.219212] acpiphp: Slot [11] registered [ 0.220168] acpiphp: Slot [12] registered [ 0.221118] acpiphp: Slot [13] registered [ 0.222137] acpiphp: Slot [14] registered [ 0.223134] acpiphp: Slot [15] registered [ 0.224368] acpiphp: Slot [16] registered [ 0.225142] acpiphp: Slot [17] registered [ 0.226124] acpiphp: Slot [18] registered [ 0.227119] acpiphp: Slot [19] registered [ 0.228223] acpiphp: Slot [20] registered [ 0.229206] acpiphp: Slot [21] registered [ 0.230221] acpiphp: Slot [22] registered [ 0.231403] acpiphp: Slot [23] registered [ 0.232311] acpiphp: Slot [24] registered [ 0.233307] acpiphp: Slot [25] registered [ 0.234229] acpiphp: Slot [26] registered [ 0.235142] acpiphp: Slot [27] registered [ 0.236106] acpiphp: Slot [28] registered [ 0.237119] acpiphp: Slot [29] registered [ 0.238187] acpiphp: Slot [30] registered [ 0.239126] acpiphp: Slot [31] registered [ 0.240051] PCI host bridge to bus 0000:00 [ 0.241020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.242188] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.243022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.244023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.245216] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.246025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.247161] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.249200] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.251709] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.258011] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.261048] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.262020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.263015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.264014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.266447] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.268478] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.269045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.271933] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.274021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.283016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.286016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.293377] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.296018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.299017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.306016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.314599] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.317027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.320024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.327016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.336081] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.339016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.342033] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.349018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.355987] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.358027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.361018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.368021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.377299] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.380017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.383016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.390019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.396359] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.399021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.402019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.409027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.416436] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.418679] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.421827] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.424737] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.428321] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.435125] iommu: Default domain type: Passthrough [ 0.436965] SCSI subsystem initialized [ 0.438239] ACPI: bus type USB registered [ 0.442168] usbcore: registered new interface driver usbfs [ 0.444137] usbcore: registered new interface driver hub [ 0.447084] usbcore: registered new device driver usb [ 0.450172] pps_core: LinuxPPS API ver. 1 registered [ 0.452717] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.456072] PTP clock support registered [ 0.459303] EDAC MC: Ver: 3.0.0 [ 0.461288] PCI: Using ACPI for IRQ routing [ 0.463000] NetLabel: Initializing [ 0.463000] NetLabel: domain hash size = 128 [ 0.463000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.467079] NetLabel: unlabeled traffic allowed by default [ 0.470641] vgaarb: loaded [ 0.473106] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.475013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.481510] clocksource: Switched to clocksource kvm-clock [ 0.674461] VFS: Disk quotas dquot_6.6.0 [ 0.675994] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.679769] *** VALIDATE ramfs *** [ 0.680985] *** VALIDATE hugetlbfs *** [ 0.684982] pnp: PnP ACPI init [ 0.689719] pnp: PnP ACPI: found 6 devices [ 0.715710] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.719080] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.721291] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.723594] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.725960] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.728331] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.731316] NET: Registered protocol family 2 [ 0.734046] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.739844] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.743819] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.749340] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.752885] TCP: Hash tables configured (established 65536 bind 65536) [ 0.756062] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.759348] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.762716] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.766145] NET: Registered protocol family 1 [ 0.768988] RPC: Registered named UNIX socket transport module. [ 0.771353] RPC: Registered udp transport module. [ 0.773326] RPC: Registered tcp transport module. [ 0.775073] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.777630] NET: Registered protocol family 44 [ 0.779533] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.781588] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.783929] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.786616] PCI: CLS 0 bytes, default 64 [ 0.788553] Unpacking initramfs... [ 2.959305] debug: unmapping init [mem 0xffff89223cc54000-0xffff89223ffbffff] [ 2.963789] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.966657] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.969792] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.535228] Initialise system trusted keyrings [ 3.537211] Key type blacklist registered [ 3.542749] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.713639] zbud: loaded [ 3.726754] *** VALIDATE nfs *** [ 3.746694] *** VALIDATE nfs4 *** [ 3.751928] pstore: using deflate compression [ 3.759246] Platform Keyring initialized [ 3.961404] NET: Registered protocol family 38 [ 3.963579] Key type asymmetric registered [ 3.965725] Asymmetric key parser 'x509' registered [ 3.967573] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.974116] io scheduler mq-deadline registered [ 3.975513] io scheduler kyber registered [ 3.977285] io scheduler bfq registered [ 3.979111] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.982120] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.984754] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.987911] ACPI: Power Button [PWRF] [ 3.993534] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.001199] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.435673] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.450145] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.474880] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.511648] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.557422] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.566585] Non-volatile memory driver v1.3 [ 4.569864] Linux agpgart interface v0.103 [ 4.636819] virtio_blk virtio1: [vda] 133608 512-byte logical blocks (68.4 MB/65.2 MiB) [ 4.642180] vda: detected capacity change from 0 to 68407296 [ 4.673798] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.698454] vdb: detected capacity change from 0 to 1073741824 [ 4.763785] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.769846] vdc: detected capacity change from 0 to 2621440000 [ 4.855137] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.863719] vdd: detected capacity change from 0 to 2621440000 [ 4.906984] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.919702] vde: detected capacity change from 0 to 4294967296 [ 4.987967] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.997819] vdf: detected capacity change from 0 to 4294967296 [ 5.086169] libphy: Fixed MDIO Bus: probed [ 5.122723] usbcore: registered new interface driver usbserial_generic [ 5.137885] usbserial: USB Serial support registered for generic [ 5.146294] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.159905] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.167454] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.181989] mousedev: PS/2 mouse device common for all mice [ 5.200386] rtc_cmos 00:05: RTC can wake from S4 [ 5.217086] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.238871] rtc_cmos 00:05: registered as rtc0 [ 5.241442] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.245623] intel_pstate: CPU model not supported [ 5.249250] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.254302] hid: raw HID events driver (C) Jiri Kosina [ 5.259692] usbcore: registered new interface driver usbhid [ 5.266683] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.271010] usbhid: USB HID core driver [ 5.273264] drop_monitor: Initializing network drop monitor service [ 5.276110] Initializing XFRM netlink socket [ 5.278950] NET: Registered protocol family 10 [ 5.295357] Segment Routing with IPv6 [ 5.297994] NET: Registered protocol family 17 [ 5.305047] mpls_gso: MPLS GSO support [ 5.315297] RAS: Correctable Errors collector initialized. [ 5.317253] AVX version of gcm_enc/dec engaged. [ 5.318917] AES CTR mode by8 optimization enabled [ 5.411704] sched_clock: Marking stable (5411533561, 0)->(7060785906, -1649252345) [ 5.415845] registered taskstats version 1 [ 5.417649] Loading compiled-in X.509 certificates [ 5.419690] zswap: loaded using pool lzo/zbud [ 5.452347] Key type big_key registered [ 5.465508] Key type encrypted registered [ 5.467162] ima: No TPM chip found, activating TPM-bypass! [ 5.469228] ima: Allocated hash algorithm: sha1 [ 5.470853] ima: No architecture policies found [ 5.473897] evm: Initialising EVM extended attributes: [ 5.475716] evm: security.selinux [ 5.476832] evm: security.ima [ 5.477779] evm: security.capability [ 5.478892] evm: HMAC attrs: 0x1 [ 5.481470] rtc_cmos 00:05: setting system clock to 2026-03-11 13:14:24 UTC (1773234864) [ 5.489145] debug: unmapping init [mem 0xffffffffb0203000-0xffffffffb03fffff] [ 5.492173] debug: unmapping init [mem 0xffffffffaef82000-0xffffffffaf258fff] [ 5.499185] Write protecting the kernel read-only data: 28672k [ 5.502966] debug: unmapping init [mem 0xffffffffad603000-0xffffffffad7fffff] [ 5.505657] debug: unmapping init [mem 0xffffffffadf14000-0xffffffffadffffff] [ 5.545684] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.554671] systemd[1]: Detected virtualization kvm. [ 5.556971] systemd[1]: Detected architecture x86-64. [ 5.559170] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.587434] systemd[1]: No hostname configured. [ 5.589139] systemd[1]: Set hostname to . [ 5.591873] random: systemd: uninitialized urandom read (16 bytes read) [ 5.595193] systemd[1]: Initializing machine ID from random generator. [ 5.787260] random: ln: uninitialized urandom read (6 bytes read) [ 5.947294] random: systemd: uninitialized urandom read (16 bytes read) [ 5.950613] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.956920] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.963122] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.586627] device-mapper: uevent: version 1.0.3 [ 7.588889] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 11.806674] random: fast init done [ 11.831228] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 13.112219] scsi host0: ata_piix [ 13.141629] scsi host1: ata_piix [ 13.146025] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.153261] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.138067] random: crng init done [ 18.184826] random: 7 urandom warning(s) missed due to ratelimiting [ 20.838705] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 23.513552] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.172449] printk: systemd: 26 output lines suppressed due to ratelimiting [ 27.214892] SELinux: Disabled at runtime. [ 27.369572] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 27.398827] systemd[1]: Detected virtualization kvm. [ 27.410830] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 29.443662] systemd[1]: initrd-switch-root.service: Succeeded. [ 29.457625] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 29.475827] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 29.490581] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 29.505212] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 29.539916] systemd[1]: Starting Journal Service... Starting Journal Service... [ 29.558152] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK [0[ 30.378098] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 31.713251] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 33.027724] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.138913] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.871170] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.946569] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit)[ 40.867445] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 41.904400] NFS: Registering the id_resolver key type [ 41.910727] Key type id_resolver registered [ 41.918205] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg442-server login: [ 95.674889] libcfs: loading out-of-tree module taints kernel. [ 95.706703] Key type ._llcrypt registered [ 95.708844] Key type .llcrypt registered [ 95.785937] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 110.051451] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 111.093134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 111.106274] alg: No test for adler32 (adler32-zlib) [ 112.321971] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 112.916692] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 114.696607] Key type lgssc registered [ 115.917094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 117.081014] hrtimer: interrupt took 3005212 ns [ 129.055165] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 129.993826] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 135.745843] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 140.486183] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 146.093646] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 156.435617] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 163.942352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 163.985945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 163.998866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 165.145147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 165.164265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 165.215807] Lustre: lustre-MDT0000: new disk, initializing [ 165.262771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 165.271909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 167.169493] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 175.775302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 175.811412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 175.855168] Lustre: 6513:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 175.874425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 175.880243] Lustre: Skipped 1 previous similar message [ 175.929349] Lustre: lustre-MDT0001: new disk, initializing [ 175.969272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 175.985305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 175.990221] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 178.317606] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.581159] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 187.194417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 187.237440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 187.390753] Lustre: lustre-OST0000: new disk, initializing [ 187.394776] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 187.439075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.291443] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.089639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 194.095981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 194.175062] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 200.586089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 200.617040] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 200.677600] Lustre: lustre-OST0001: new disk, initializing [ 200.681290] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 200.721242] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.029564] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.896890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 207.913499] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 207.946471] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 213.851562] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 225.184342] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 233.226288] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing check_logdir /tmp/testlogs/ [ 236.086924] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing yml_node [ 238.197876] Lustre: DEBUG MARKER: Client: 2.16.59.50 [ 239.811556] Lustre: DEBUG MARKER: MDS: 2.16.59.50 [ 241.457061] Lustre: DEBUG MARKER: OSS: 2.16.59.50 [ 242.500520] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Mar 11 09:18:21 EDT 2026 [ 252.807902] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 253.696658] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 268.256050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 268.264645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 268.273647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 269.279946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 269.280434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 269.289609] Lustre: Skipped 3 previous similar messages [ 272.789208] LustreError: 12214:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 272.874949] Lustre: server umount lustre-MDT0000 complete [ 275.050799] LustreError: 6503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235134 with bad export cookie 11675504480320666009 [ 275.051067] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 275.058104] LustreError: 6503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 279.519849] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.522647] LustreError: 12114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 279.522871] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 279.526924] Lustre: Skipped 3 previous similar messages [ 279.542070] LustreError: 12114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 280.544926] LustreError: 8426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.568852] LustreError: 8426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 281.237492] LustreError: 12417:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 281.242686] LustreError: 12417:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 281.309327] Lustre: server umount lustre-MDT0001 complete [ 283.501693] LustreError: 12618:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 283.505468] LustreError: 12618:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 283.554596] Lustre: server umount lustre-OST0000 complete [ 285.916886] LustreError: 12818:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 285.920132] LustreError: 12818:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 285.995442] Lustre: server umount lustre-OST0001 complete [ 290.284699] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 296.187310] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 301.938710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 305.512504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 309.070208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 313.197192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 318.131771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 318.165344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 318.382272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 318.417756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 318.490589] Lustre: lustre-MDT0000: new disk, initializing [ 318.559510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 318.572613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 321.128414] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 328.430279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 328.507366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 328.554291] Lustre: 14998:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 328.609204] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 328.613733] Lustre: Skipped 1 previous similar message [ 328.689587] Lustre: lustre-MDT0001: new disk, initializing [ 328.756987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 328.814658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 328.828647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 331.244418] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 336.350465] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 338.968989] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 343.181654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 343.225279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 345.134128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 345.139716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 345.208145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 346.588256] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 351.811529] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 354.668953] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 354.819523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 357.632957] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 357.788348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 360.433261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.441532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 360.441607] Lustre: Skipped 1 previous similar message [ 360.449729] Lustre: Skipped 4 previous similar messages [ 365.263412] LustreError: 17941:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 365.266089] LustreError: 17941:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 365.430305] Lustre: server umount lustre-OST0000 complete [ 369.632766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 369.644358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 369.656855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 374.069224] LustreError: 18143:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 374.071443] LustreError: 18143:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 374.189491] Lustre: server umount lustre-MDT0000 complete [ 375.776860] LustreError: 15012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 375.786833] LustreError: 15012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 376.697625] LustreError: 14992:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235235 with bad export cookie 11675504480320667542 [ 376.699118] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 376.704281] LustreError: 14992:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 376.895877] Lustre: server umount lustre-MDT0001 complete [ 379.352825] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:20:38 (1773235238) [ 380.892429] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 382.047035] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:20:40 (1773235240) [ 385.937553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.216642] LustreError: 19014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 386.251647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 386.255027] Lustre: Skipped 1 previous similar message [ 388.241528] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 391.656720] LustreError: 19014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 393.584782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 395.930134] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 398.981735] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 401.269981] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 405.265633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 408.219405] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 411.267165] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 426.033098] LustreError: 21279:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 426.036117] LustreError: 21279:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 426.097913] Lustre: server umount lustre-OST0000 complete [ 429.025898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 429.032570] Lustre: Skipped 1 previous similar message [ 429.034842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 429.038265] Lustre: Skipped 3 previous similar messages [ 442.335128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 442.481186] Lustre: server umount lustre-MDT0000 complete [ 444.385697] LustreError: 19008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.398318] LustreError: 19008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 444.732927] LustreError: 19039:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235303 with bad export cookie 11675504480320668424 [ 444.739128] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 444.741180] LustreError: 19039:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 454.186448] Lustre: server umount lustre-MDT0001 complete [ 461.303692] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 462.969296] Key type lgssc unregistered [ 463.138746] LNet: 22216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 463.143831] LNetError: 22216:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 463.162088] LNet: Removed LNI 192.168.204.142@tcp [ 463.723229] Key type .llcrypt unregistered [ 463.724677] Key type ._llcrypt unregistered [ 476.466823] Key type ._llcrypt registered [ 476.468924] Key type .llcrypt registered [ 476.529827] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 477.050835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 477.106949] alg: No test for adler32 (adler32-zlib) [ 478.026088] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 478.144828] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 479.735202] Key type lgssc registered [ 480.318663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 484.811057] loop: module loaded [ 484.852720] loop0: detected capacity change from 0 to 204800000 [ 485.035018] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 485.045794] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 485.214882] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 485.250765] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 488.650297] loop0: detected capacity change from 0 to 204800000 [ 488.728305] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 488.733155] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 488.740654] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 488.803474] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 491.533153] loop0: detected capacity change from 0 to 204800000 [ 495.182038] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 495.219846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 495.232694] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 496.415917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 496.469652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 496.524327] Lustre: lustre-MDT0000: new disk, initializing [ 496.576246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.591960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 496.612731] Lustre: 24328:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 466, rollback = 2 [ 496.618804] Lustre: 24328:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 496.624254] Lustre: 24328:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 2/466/0 [ 496.628729] Lustre: 24328:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 496.633327] Lustre: 24328:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 496.636267] Lustre: 24328:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 498.887430] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 504.624868] loop1: detected capacity change from 0 to 204800000 [ 507.919270] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 507.946200] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 508.074259] Lustre: lustre-OST07c6: new disk, initializing [ 508.077425] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 508.079751] Lustre: Skipped 1 previous similar message [ 508.112097] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 511.643585] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 516.128208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 516.134912] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 516.161502] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 519.772399] Lustre: 24354:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 519.779600] Lustre: 24354:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 1 previous similar message [ 519.782765] Lustre: 24354:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 519.786249] Lustre: 24354:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 519.790324] Lustre: 24354:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 519.797096] Lustre: 24354:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 519.801390] Lustre: 24354:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 519.805719] Lustre: 24354:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 519.810816] Lustre: 24354:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 519.814304] Lustre: 24354:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 519.818265] Lustre: 24354:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 519.822531] Lustre: 24354:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 538.254203] Lustre: 24353:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 497, rollback = 2 [ 538.257148] Lustre: 24353:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 538.259524] Lustre: 24353:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 538.262063] Lustre: 24353:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 538.265977] Lustre: 24353:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 538.269171] Lustre: 24353:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 542.270412] Lustre: 24352:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 497, rollback = 2 [ 542.275238] Lustre: 24352:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 1 previous similar message [ 542.279515] Lustre: 24352:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 542.283145] Lustre: 24352:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 542.287245] Lustre: 24352:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 542.290982] Lustre: 24352:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 542.294646] Lustre: 24352:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 542.300479] Lustre: 24352:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 542.306694] Lustre: 24352:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 542.311191] Lustre: 24352:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 542.315923] Lustre: 24352:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 542.320061] Lustre: 24352:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 549.415514] Lustre: 24354:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 497, rollback = 2 [ 549.423797] Lustre: 24354:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 3 previous similar messages [ 549.429218] Lustre: 24354:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 549.434036] Lustre: 24354:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 549.438965] Lustre: 24354:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 549.443669] Lustre: 24354:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 549.450655] Lustre: 24354:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 549.457938] Lustre: 24354:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 549.465903] Lustre: 24354:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 549.469560] Lustre: 24354:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 549.474261] Lustre: 24354:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 549.477558] Lustre: 24354:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 560.626875] Lustre: 25942:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 497, rollback = 2 [ 560.636843] Lustre: 25942:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 5 previous similar messages [ 560.643216] Lustre: 25942:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 560.647222] Lustre: 25942:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 560.652826] Lustre: 25942:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 560.656865] Lustre: 25942:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 560.661901] Lustre: 25942:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 560.666783] Lustre: 25942:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 560.673583] Lustre: 25942:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 560.679546] Lustre: 25942:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 560.684800] Lustre: 25942:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 560.689471] Lustre: 25942:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 607.053230] LustreError: 26559:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 607.100385] Lustre: server umount lustre-OST07c6 complete [ 621.033248] LustreError: 27059:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 621.035816] LustreError: 27059:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 621.149213] Lustre: server umount lustre-MDT0000 complete [ 629.419017] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 630.810370] Key type lgssc unregistered [ 630.975742] LNet: 27894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.981452] LNetError: 27894:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.990550] LNet: Removed LNI 192.168.204.142@tcp [ 631.413519] Key type .llcrypt unregistered [ 631.415612] Key type ._llcrypt unregistered [ 640.591319] Key type ._llcrypt registered [ 640.592558] Key type .llcrypt registered [ 640.660284] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 641.115733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 641.179253] alg: No test for adler32 (adler32-zlib) [ 642.079250] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 642.213704] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 643.815200] Key type lgssc registered [ 644.414537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 648.751739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 648.768525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.026760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 652.231585] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 656.966984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 657.163215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 659.201850] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 662.147470] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 664.058539] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 667.483072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 667.676829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 670.686463] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 673.424770] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 675.882295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 686.989960] LustreError: 31345:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 692.194996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 692.195950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 692.210130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 693.245245] LustreError: 31596:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 693.318682] Lustre: server umount lustre-OST0000 complete [ 697.314979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 697.317323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 697.322590] Lustre: Skipped 2 previous similar messages [ 697.325144] Lustre: Skipped 1 previous similar message [ 701.391410] LustreError: 31797:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 701.395337] LustreError: 31797:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 701.431674] Lustre: server umount lustre-MDT0000 complete [ 702.432250] LustreError: 29126:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 702.439672] LustreError: 29126:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 703.312767] LustreError: 29107:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235562 with bad export cookie 1902324441330696712 [ 703.316599] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 703.318052] LustreError: 29107:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 703.415734] LustreError: 31999:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 703.417954] LustreError: 31999:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 703.490556] Lustre: server umount lustre-MDT0001 complete [ 708.059164] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 709.551798] Key type lgssc unregistered [ 709.729810] LNet: 32531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.733229] LNetError: 32531:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 709.743484] LNet: Removed LNI 192.168.204.142@tcp [ 710.128188] Key type .llcrypt unregistered [ 710.129764] Key type ._llcrypt unregistered [ 720.121599] Key type ._llcrypt registered [ 720.122705] Key type .llcrypt registered [ 720.192632] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 720.722169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 720.735943] alg: No test for adler32 (adler32-zlib) [ 721.631182] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 721.760528] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 723.359419] Key type lgssc registered [ 723.904210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 728.168467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 728.175727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 729.445699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 731.374879] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 736.001993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 736.234884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 738.544945] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 741.705407] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 743.621568] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 747.151839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 747.340789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 750.029865] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 752.561996] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 754.536943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 770.016398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 770.022724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 770.035064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 772.065777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 772.066737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 772.070612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 774.714150] LustreError: 36024:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 774.784580] Lustre: server umount lustre-OST0000 complete [ 777.184584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 777.187044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 777.188837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 777.193841] Lustre: Skipped 1 previous similar message [ 777.199103] Lustre: Skipped 2 previous similar messages [ 782.304570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 782.307210] Lustre: Skipped 1 previous similar message [ 782.792714] LustreError: 36225:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 782.795878] LustreError: 36225:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 782.867323] Lustre: server umount lustre-MDT0000 complete [ 784.957353] LustreError: 35378:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235643 with bad export cookie 14336650363620496636 [ 784.961414] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 784.967169] LustreError: 35378:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 785.069743] LustreError: 36429:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 785.074573] LustreError: 36429:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 785.199398] Lustre: server umount lustre-MDT0001 complete [ 790.579493] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 792.170990] Key type lgssc unregistered [ 792.351706] LNet: 36960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.356130] LNetError: 36960:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.367802] LNet: Removed LNI 192.168.204.142@tcp [ 792.799325] Key type .llcrypt unregistered [ 792.800856] Key type ._llcrypt unregistered [ 805.234903] Key type ._llcrypt registered [ 805.237177] Key type .llcrypt registered [ 805.302087] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 805.863229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 805.932455] alg: No test for adler32 (adler32-zlib) [ 806.852274] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 806.975196] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 808.607154] Key type lgssc registered [ 809.185287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 813.059468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 813.068426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 814.341665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 816.393056] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 821.060470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 821.320024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 823.261572] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 826.089955] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 827.846483] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 831.218543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 831.386728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 834.261359] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 837.272288] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 839.598983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 846.822053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 846.827043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 846.834788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 847.844216] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.42@tcp (stopping) [ 849.889470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 849.891748] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 849.898917] Lustre: Skipped 1 previous similar message [ 851.484187] LustreError: 40578:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 851.543898] Lustre: server umount lustre-MDT0000 complete [ 852.953632] LustreError: 38329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 852.966867] LustreError: 38329:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 853.668497] LustreError: 39952:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235712 with bad export cookie 5805093973577797120 [ 853.674189] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 853.675789] LustreError: 39952:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 855.008134] LustreError: 38329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 855.008665] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.025770] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 855.030344] Lustre: Skipped 2 previous similar messages [ 858.016624] LustreError: 38330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 859.864518] LustreError: 40781:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 859.868056] LustreError: 40781:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 859.945190] Lustre: server umount lustre-MDT0001 complete [ 887.266012] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 887.295437] LustreError: 41004:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 887.298459] LustreError: 41004:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 887.365907] Lustre: server umount lustre-OST0000 complete [ 896.615183] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 898.132693] Key type lgssc unregistered [ 898.308282] LNet: 41842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.313288] LNetError: 41842:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.323914] LNet: Removed LNI 192.168.204.142@tcp [ 898.819905] Key type .llcrypt unregistered [ 898.821426] Key type ._llcrypt unregistered [ 911.314861] Key type ._llcrypt registered [ 911.316526] Key type .llcrypt registered [ 911.364511] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 911.955801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 911.964061] alg: No test for adler32 (adler32-zlib) [ 912.858683] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 912.995313] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 914.607140] Key type lgssc registered [ 915.087093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 918.495792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 918.506510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 919.734110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 921.429116] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 925.444575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 925.612525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 927.597459] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 930.336510] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 932.076144] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 935.409468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 935.629278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 938.700912] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 941.537932] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 944.872793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 952.288952] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 952.293320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 952.299174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.316971] Lustre: lustre-OST0000: Not available for connect from 192.168.204.42@tcp (stopping) [ 955.362262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 955.366150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.367161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 955.376559] Lustre: Skipped 1 previous similar message [ 956.780095] LustreError: 45462:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 956.823177] Lustre: server umount lustre-OST0000 complete [ 971.743990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 971.751215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.756088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 975.842545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 975.842993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 985.056275] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 985.056653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 985.071986] Lustre: Skipped 4 previous similar messages [ 985.105479] LustreError: 45840:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 985.112496] LustreError: 45840:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 985.154352] Lustre: server umount lustre-MDT0000 complete [ 987.362114] LustreError: 43198:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235846 with bad export cookie 1338878700321693497 [ 987.377104] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 993.557963] LustreError: 46043:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 993.567381] LustreError: 46043:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 993.672604] Lustre: server umount lustre-MDT0001 complete [ 999.497306] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1001.077330] Key type lgssc unregistered [ 1001.245869] LNet: 46575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.250258] LNetError: 46575:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1001.259665] LNet: Removed LNI 192.168.204.142@tcp [ 1001.671867] Key type .llcrypt unregistered [ 1001.673635] Key type ._llcrypt unregistered [ 1013.312108] Key type ._llcrypt registered [ 1013.313789] Key type .llcrypt registered [ 1013.368490] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1013.817734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1013.861420] alg: No test for adler32 (adler32-zlib) [ 1014.773580] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1014.903364] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1016.519169] Key type lgssc registered [ 1017.284957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1021.423434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1021.431300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1022.647455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1024.464719] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1028.981214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1029.216379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1031.415221] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1034.148756] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1035.723415] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1038.903566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1039.128359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.764330] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1044.002509] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1047.339299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1052.804037] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1054.522443] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1065.503389] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-03-11 9h32m04s [ 1066.155416] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1068.002136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.006341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1068.007638] Lustre: Skipped 2 previous similar messages [ 1068.007848] LustreError: 47264:0:(import.c:716:ptlrpc_connect_import_locked()) already connecting [ 1068.011180] Lustre: Skipped 2 previous similar messages [ 1069.029921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1071.074450] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.42@tcp (stopping) [ 1071.076812] Lustre: Skipped 2 previous similar messages [ 1075.169980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1075.174177] Lustre: Skipped 3 previous similar messages [ 1081.311152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1081.314044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1081.322529] Lustre: Skipped 3 previous similar messages [ 1081.358801] LustreError: 50518:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1081.393361] Lustre: server umount lustre-MDT0000 complete [ 1082.906989] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1083.354612] LustreError: 47947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1083.365667] LustreError: 47947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1085.944950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1086.012457] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1086.113301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1086.139424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1087.144822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1088.059428] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1089.676914] Lustre: DEBUG MARKER: Wait for df (36066) ... [ 1090.361742] Lustre: DEBUG MARKER: done [ 1092.577993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.579849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1092.583390] Lustre: Skipped 1 previous similar message [ 1097.713118] LustreError: 51566:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1097.718748] LustreError: 51566:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1097.762441] Lustre: server umount lustre-OST0000 complete [ 1102.815820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1102.818567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1105.536612] LustreError: 51772:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1105.539994] LustreError: 51772:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1105.569049] Lustre: server umount lustre-MDT0000 complete [ 1107.103514] LustreError: 50249:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235966 with bad export cookie 620944970483509387 [ 1107.106267] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1107.108478] LustreError: 50249:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1107.259732] Lustre: server umount lustre-MDT0001 complete [ 1111.362249] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1112.534730] Key type lgssc unregistered [ 1112.682438] LNet: 52503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.685038] LNetError: 52503:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1112.691604] LNet: Removed LNI 192.168.204.142@tcp [ 1113.037940] Key type .llcrypt unregistered [ 1113.039637] Key type ._llcrypt unregistered [ 1127.592697] Key type ._llcrypt registered [ 1127.594207] Key type .llcrypt registered [ 1127.664798] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 1135.645247] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1136.024336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1136.061052] alg: No test for adler32 (adler32-zlib) [ 1136.970105] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1137.056757] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1138.647164] Key type lgssc registered [ 1139.028833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1142.511338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1145.146024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1147.770191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1150.259281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1153.088672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1153.121234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1153.132089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1154.234438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1154.254388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1154.300376] Lustre: lustre-MDT0000: new disk, initializing [ 1154.340681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.348497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1155.930097] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1161.069441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1161.089935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1161.144392] Lustre: 56438:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1161.189682] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1161.192104] Lustre: Skipped 1 previous similar message [ 1161.238238] Lustre: lustre-MDT0001: new disk, initializing [ 1161.289328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1161.322136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1161.327753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1163.017854] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1166.807606] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1168.270534] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1171.130273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1171.169206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1171.295297] Lustre: lustre-OST0000: new disk, initializing [ 1171.298266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1171.364442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1173.735836] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1176.574563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1176.578876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1176.625833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1177.339728] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1178.951218] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1179.029030] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1180.453798] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1180.524164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1181.668791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.668809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1181.669473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1181.674265] Lustre: Skipped 1 previous similar message [ 1181.677063] LustreError: Skipped 1 previous similar message [ 1183.214081] LustreError: 59376:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1183.257804] Lustre: server umount lustre-OST0000 complete [ 1186.784822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1186.785060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1186.791463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1186.797310] Lustre: Skipped 2 previous similar messages [ 1191.905325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1191.907901] Lustre: Skipped 1 previous similar message [ 1197.024838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1197.027233] Lustre: Skipped 1 previous similar message [ 1199.071105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1199.126941] LustreError: 59576:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1199.129661] LustreError: 59576:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1199.169432] Lustre: server umount lustre-MDT0000 complete [ 1200.679743] LustreError: 57216:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236059 with bad export cookie 13420605917093203315 [ 1200.683511] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1200.685482] LustreError: 57216:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1200.752459] LustreError: 59781:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1200.754742] LustreError: 59781:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1200.809177] Lustre: server umount lustre-MDT0001 complete [ 1204.138675] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 09:34:23 (1773236063) [ 1207.061613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1207.267057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.837550] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1212.387962] LustreError: 60451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.395532] LustreError: 60451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1212.871942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1213.046198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1214.805881] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1217.460384] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1219.269169] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1222.621570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1222.795500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.855302] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1228.598842] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1236.241510] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-03-11 9h34m55s [ 1236.822379] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1239.557688] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1240.583716] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1240.600324] LustreError: 60447:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1261.029774] Lustre: lustre-MDT0000: Client d28ea974-c377-4810-9ed1-fec61a12cfad (at 192.168.204.42@tcp) reconnecting [ 1280.647125] LustreError: 60447:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1281.437210] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1282.311192] Lustre: DEBUG MARKER: done [ 1285.599909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1285.603197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.608036] Lustre: Skipped 1 previous similar message [ 1285.610332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1290.230102] LustreError: 63453:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1290.232395] LustreError: 63453:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1290.284912] Lustre: server umount lustre-OST0000 complete [ 1293.793329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.793811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1293.797355] Lustre: Skipped 2 previous similar messages [ 1293.800162] Lustre: Skipped 3 previous similar messages [ 1306.079130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1306.126084] LustreError: 63655:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1306.131379] LustreError: 63655:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1306.156274] Lustre: server umount lustre-MDT0000 complete [ 1307.545260] LustreError: 62836:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236166 with bad export cookie 13420605917093204197 [ 1307.549978] LustreError: 62836:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1307.550104] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1307.711958] Lustre: server umount lustre-MDT0001 complete [ 1311.553515] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1312.645689] Key type lgssc unregistered [ 1312.783353] LNet: 64390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1312.786636] LNetError: 64390:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1312.795539] LNet: Removed LNI 192.168.204.142@tcp [ 1313.112851] Key type .llcrypt unregistered [ 1313.114306] Key type ._llcrypt unregistered [ 1324.967413] Key type ._llcrypt registered [ 1324.969133] Key type .llcrypt registered [ 1325.017706] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 1331.448611] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1331.884571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1331.896487] alg: No test for adler32 (adler32-zlib) [ 1332.754496] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1332.842747] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1334.431148] Key type lgssc registered [ 1334.824804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1338.698620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1341.275886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1343.839074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1346.597420] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1349.188792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1349.209327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1349.214641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.312166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1350.326859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1350.372188] Lustre: lustre-MDT0000: new disk, initializing [ 1350.412218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.421132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1352.015782] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1356.802536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1356.843602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1356.880345] Lustre: 68315:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1356.908894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1356.911227] Lustre: Skipped 1 previous similar message [ 1356.947523] Lustre: lustre-MDT0001: new disk, initializing [ 1356.982301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1356.996181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1357.000101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1358.431601] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1361.746251] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1363.123336] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1365.843744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1365.870210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1365.972891] Lustre: lustre-OST0000: new disk, initializing [ 1365.974943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1366.006232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1368.144172] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1371.624794] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1373.130823] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1374.204838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1374.209581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1374.239269] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1375.270345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 1376.892575] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1376.976751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1379.298467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1379.300630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1379.302120] LustreError: Skipped 1 previous similar message [ 1379.307785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1379.311173] Lustre: Skipped 1 previous similar message [ 1383.855113] LustreError: 71268:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1383.894235] Lustre: server umount lustre-OST0000 complete [ 1387.999867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1388.003352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1388.008522] Lustre: Skipped 1 previous similar message [ 1388.011294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1389.537473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1389.540250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1391.489994] LustreError: 71469:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1391.492806] LustreError: 71469:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1391.525216] Lustre: server umount lustre-MDT0000 complete [ 1392.952057] LustreError: 68305:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236251 with bad export cookie 2218207680092260562 [ 1392.953687] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1392.956703] LustreError: 68305:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1393.019418] LustreError: 71670:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1393.021559] LustreError: 71670:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1393.076782] Lustre: server umount lustre-MDT0001 complete [ 1396.173578] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 09:37:35 (1773236255) [ 1396.685877] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1397.315202] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 09:37:36 (1773236256) [ 1402.063987] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1403.839243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1404.118418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.571634] Lustre: Failing over lustre-MDT0000 [ 1404.574525] LustreError: 72744:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1404.580812] LustreError: 72744:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1404.588944] LustreError: 72744:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1404.673142] LustreError: 72795:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1404.677341] LustreError: 72795:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1404.736834] Lustre: server umount lustre-MDT0000 complete [ 1407.296463] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:37:46 (1773236266) [ 1410.080091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.276343] LustreError: 73362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1410.286261] LustreError: 73362:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1410.299407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1411.857382] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1415.179344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1416.787806] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1419.025750] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1420.486331] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1423.171985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1423.316611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1423.320015] Lustre: Skipped 1 previous similar message [ 1425.623986] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1427.614335] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1440.490053] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1445.858728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.861701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1445.864491] Lustre: Skipped 1 previous similar message [ 1445.868193] Lustre: Skipped 3 previous similar messages [ 1448.449313] LustreError: 75724:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1448.451376] LustreError: 75724:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1448.483236] Lustre: server umount lustre-MDT0000 complete [ 1449.988250] LustreError: 73343:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236309 with bad export cookie 2218207680092261703 [ 1449.990146] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1449.993454] LustreError: 73343:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1450.144293] Lustre: server umount lustre-MDT0001 complete [ 1451.583399] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1453.700553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1457.406327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1460.192683] LustreError: 66525:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006590c3e8 x1859372912273152/t0(0) o250->MGC192.168.204.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1460.321710] LustreError: 74968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1460.351919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.393100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1461.967522] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1465.425248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1465.505050] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1465.506494] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1465.507739] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1465.515870] Lustre: Skipped 1 previous similar message [ 1466.847513] Lustre: 66528:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773236309/real 1773236309] req@000000000a093860 x1859372912272896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773236325 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1467.216496] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1469.640313] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1470.948965] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1471.002995] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1471.967560] Lustre: 66529:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773236314/real 1773236314] req@0000000003a18c84 x1859372912273024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773236330 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1476.066042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1476.067308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.072126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1476.076151] Lustre: Skipped 2 previous similar messages [ 1476.079288] Lustre: Skipped 2 previous similar messages [ 1477.087205] Lustre: 66526:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773236319/real 1773236319] req@0000000043c8d142 x1859372912273280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773236335 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1479.235737] LustreError: 78184:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1479.237981] LustreError: 78184:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1479.266206] Lustre: server umount lustre-MDT0000 complete [ 1480.678232] LustreError: 73342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236339 with bad export cookie 2218207680092263593 [ 1480.681182] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1480.682457] LustreError: 73342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1480.790299] Lustre: server umount lustre-MDT0001 complete [ 1483.711331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1487.743966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1490.335923] LustreError: 66525:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b3cd0581 x1859372912291456/t0(0) o250->MGC192.168.204.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1490.476446] LustreError: 74967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1490.533255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1491.863310] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1495.093639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1495.200729] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1495.244687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1495.247011] Lustre: Skipped 2 previous similar messages [ 1496.293203] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1496.600580] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1497.311159] Lustre: 66529:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773236340/real 1773236340] req@00000000058830ef x1859372912291072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773236356 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1498.745894] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1500.252634] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1501.663207] Lustre: 66529:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773236344/real 1773236344] req@00000000b5c79990 x1859372912291200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1773236360 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1505.760316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1505.765670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1508.350894] LustreError: 80642:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1508.353967] LustreError: 80642:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1508.383821] Lustre: server umount lustre-MDT0000 complete [ 1509.855415] LustreError: 75725:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236368 with bad export cookie 2218207680092264965 [ 1509.859242] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1509.860300] LustreError: 75725:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1509.970988] Lustre: server umount lustre-MDT0001 complete [ 1512.278317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1513.537411] Lustre: DEBUG MARKER: files compared the same [ 1524.581417] Lustre: server umount lustre-OST0000 complete [ 1531.433044] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1532.663551] Key type lgssc unregistered [ 1532.806745] LNet: 82274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1532.810605] LNetError: 82274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1532.817700] LNet: Removed LNI 192.168.204.142@tcp [ 1533.164766] Key type .llcrypt unregistered [ 1533.166170] Key type ._llcrypt unregistered [ 1543.829721] Key type ._llcrypt registered [ 1543.831514] Key type .llcrypt registered [ 1543.881444] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1544.287651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1544.330181] alg: No test for adler32 (adler32-zlib) [ 1545.221875] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1545.324657] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1546.927694] Key type lgssc registered [ 1547.561799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1551.013664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1551.022039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1552.226185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1553.925759] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug +malloc all [ 1557.381938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1557.516389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1559.094841] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug +malloc all [ 1561.202592] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1562.756799] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1565.910894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1566.022792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1568.327994] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug +malloc all [ 1570.946217] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1578.340510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1587.679606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1587.682899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1587.687687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1588.703941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1588.704377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1588.704626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1591.791076] LustreError: 85911:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1591.828076] Lustre: server umount lustre-OST0000 complete [ 1593.824699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1593.830529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1593.834161] Lustre: Skipped 1 previous similar message [ 1598.432148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1598.435151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1598.440649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1599.691965] LustreError: 86113:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1599.695049] LustreError: 86113:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1599.736878] Lustre: server umount lustre-MDT0000 complete [ 1601.483739] LustreError: 86114:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236460 with bad export cookie 16885771999682187945 [ 1601.484674] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1601.488971] LustreError: 86114:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1601.558772] LustreError: 86317:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1601.561107] LustreError: 86317:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1601.617457] Lustre: server umount lustre-MDT0001 complete [ 1606.069395] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1607.330062] Key type lgssc unregistered [ 1607.474493] LNet: 86848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1607.477986] LNetError: 86848:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1607.490403] LNet: Removed LNI 192.168.204.142@tcp [ 1607.893210] Key type .llcrypt unregistered [ 1607.894932] Key type ._llcrypt unregistered [ 1619.082869] Key type ._llcrypt registered [ 1619.085171] Key type .llcrypt registered [ 1619.136775] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1619.585674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1619.593322] alg: No test for adler32 (adler32-zlib) [ 1620.477054] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1620.578236] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1622.183227] Key type lgssc registered [ 1622.701915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1625.959644] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1625.971156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1641.439348] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1646.559207] LustreError: 88196:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.142@tcp: failed processing log, type 4: rc = -110 [ 1673.183299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1675.834074] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1678.095325] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1681.241980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1681.298453] Lustre: *** cfs_fail_loc=706, val=0*** [ 1681.493196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1681.542263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1683.316992] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1686.497498] LustreError: 88223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1686.506047] LustreError: 88223:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1687.342497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1687.501136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1689.084543] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1691.107801] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1692.554480] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1693.664761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1693.667290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1693.669479] Lustre: Skipped 1 previous similar message [ 1693.673770] Lustre: Skipped 1 previous similar message [ 1698.785624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1698.787984] Lustre: Skipped 1 previous similar message [ 1699.438462] LustreError: 90521:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1699.477761] Lustre: server umount lustre-OST0000 complete [ 1702.879863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1702.882438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.886542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1703.904755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1709.029519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1709.032057] Lustre: Skipped 3 previous similar messages [ 1714.144118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1714.146452] Lustre: Skipped 1 previous similar message [ 1715.167116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1715.213881] LustreError: 90723:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1715.217105] LustreError: 90723:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1715.258300] Lustre: server umount lustre-MDT0000 complete [ 1716.901200] LustreError: 88203:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236575 with bad export cookie 13100743041905305736 [ 1716.904518] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1716.906522] LustreError: 88203:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1716.984686] LustreError: 90930:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1716.987389] LustreError: 90930:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1717.035820] Lustre: server umount lustre-MDT0001 complete [ 1721.533151] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1722.925302] Key type lgssc unregistered [ 1723.073710] LNet: 91461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.077398] LNetError: 91461:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.134454] LNet: Removed LNI 192.168.204.142@tcp [ 1724.502674] Key type .llcrypt unregistered [ 1724.503813] Key type ._llcrypt unregistered [ 1735.658383] Key type ._llcrypt registered [ 1735.660715] Key type .llcrypt registered [ 1735.707147] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1736.116868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1736.160091] alg: No test for adler32 (adler32-zlib) [ 1737.022818] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1737.134489] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1738.744037] Key type lgssc registered [ 1739.295778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1742.910162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1742.917309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1745.638693] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1749.117388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1749.355263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1751.281662] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1754.598547] LustreError: 93503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1754.608655] LustreError: 93503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1755.442598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1755.615944] LustreError: 93503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1755.632385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1758.231873] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1759.714161] LustreError: 93507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1759.725630] LustreError: 93507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1762.014505] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1762.022939] Lustre: MGS: Client 6110e416-98b7-481e-a970-f5ce43d4a137 (at 0@lo) reconnecting [ 1762.144132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.186765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1763.728313] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1777.635044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1777.638822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1777.651968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1777.657322] Lustre: Skipped 1 previous similar message [ 1779.254131] LustreError: 95469:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1779.307962] Lustre: server umount lustre-OST0000 complete [ 1782.754519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.758115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1782.759277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1782.763835] Lustre: Skipped 2 previous similar messages [ 1787.207050] LustreError: 95670:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1787.209655] LustreError: 95670:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1787.221213] Lustre: server umount lustre-MDT0000 complete [ 1787.871872] LustreError: 95018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1787.878214] LustreError: 95018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1788.856825] LustreError: 95871:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1788.860219] LustreError: 95871:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1788.912723] Lustre: server umount lustre-MDT0001 complete [ 1790.532283] Lustre: server umount MGS complete [ 1794.865440] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:44:13 (1773236653) [ 1804.928185] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 1808.371783] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1812.312709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1814.613850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1816.958834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1819.188242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1821.603608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1821.625884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1822.965792] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1827.147309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1827.175484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1827.258264] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1827.280125] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1827.320142] Lustre: lustre-MDT0001: new disk, initializing [ 1827.350330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1827.360537] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1827.364718] Lustre: 100581:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1827.369518] LustreError: 100581:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1828.777701] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1833.329992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1833.356939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1833.453969] Lustre: lustre-OST0000: new disk, initializing [ 1833.456529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1833.480207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1835.671159] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1837.664133] Lustre: 101775:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@0000000071cd1bde x1859373336922368/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1773236712 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1839.066443] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1841.737882] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1841.743923] Lustre: MGS: Client c56e0749-00ca-42bf-a2ea-5bd24628d8dd (at 0@lo) reconnecting [ 1841.762297] Lustre: 99865:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1841.773037] LustreError: 99865:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1841.787103] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1841.831655] Lustre: lustre-MDT0000: new disk, initializing [ 1841.881458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1841.893525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1842.911625] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1842.915447] Lustre: 102295:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@00000000a299675e x1859373336927872/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1773236717 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1843.308206] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1844.961908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1844.965547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1845.034905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1862.111801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1862.114394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.118886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1862.121097] Lustre: Skipped 1 previous similar message [ 1864.160960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1864.163498] Lustre: Skipped 2 previous similar messages [ 1868.079243] LustreError: 102817:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1868.081333] LustreError: 102817:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1868.122922] Lustre: server umount lustre-OST0000 complete [ 1874.400741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1874.400913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1874.404433] Lustre: Skipped 1 previous similar message [ 1874.406712] LustreError: Skipped 1 previous similar message [ 1874.407272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1874.410884] Lustre: Skipped 1 previous similar message [ 1875.709635] LustreError: 103019:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1875.712993] LustreError: 103019:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1875.721798] Lustre: server umount lustre-MDT0000 complete [ 1883.241166] Lustre: server umount lustre-MDT0001 complete [ 1884.544040] LustreError: 103423:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1884.547313] LustreError: 103423:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1884.574637] Lustre: server umount MGS complete [ 1888.522982] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:45:47 (1773236747) [ 1894.718316] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1895.803346] Key type lgssc unregistered [ 1895.927343] LNet: 104757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.930827] LNetError: 104757:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1895.939383] LNet: Removed LNI 192.168.204.142@tcp [ 1896.253816] Key type .llcrypt unregistered [ 1896.255459] Key type ._llcrypt unregistered [ 1903.562605] Key type ._llcrypt registered [ 1903.563684] Key type .llcrypt registered [ 1903.601506] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1903.930810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1903.965063] alg: No test for adler32 (adler32-zlib) [ 1904.859367] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1904.954052] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1906.543182] Key type lgssc registered [ 1906.940073] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1909.861516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1909.873405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.072916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.633545] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1916.081509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1916.294261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1917.859968] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1920.114356] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1921.529369] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1924.325443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1924.467568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1926.439056] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1928.602396] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1936.746756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1941.985535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1941.986867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1941.988091] LustreError: Skipped 1 previous similar message [ 1941.988281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1941.992831] Lustre: Skipped 1 previous similar message [ 1941.998018] Lustre: Skipped 1 previous similar message [ 1947.105971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1947.108042] Lustre: Skipped 1 previous similar message [ 1952.227823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1952.230059] Lustre: Skipped 1 previous similar message [ 1956.319133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1956.339089] LustreError: 108202:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1956.378868] Lustre: server umount lustre-OST0000 complete [ 1962.464779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1962.465720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.466298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1962.475581] Lustre: Skipped 1 previous similar message [ 1963.775650] LustreError: 108405:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1963.779246] LustreError: 108405:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1963.809503] Lustre: server umount lustre-MDT0000 complete [ 1965.268149] LustreError: 105916:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236824 with bad export cookie 11811608666805938985 [ 1965.271079] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1965.272288] LustreError: 105916:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1965.337171] LustreError: 108607:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1965.339164] LustreError: 108607:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1965.395028] Lustre: server umount lustre-MDT0001 complete [ 1969.700254] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1970.946905] Key type lgssc unregistered [ 1971.085545] LNet: 109138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1971.088859] LNetError: 109138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1971.095518] LNet: Removed LNI 192.168.204.142@tcp [ 1971.467486] Key type .llcrypt unregistered [ 1971.469198] Key type ._llcrypt unregistered [ 1978.074409] Key type ._llcrypt registered [ 1978.075936] Key type .llcrypt registered [ 1978.111935] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1978.422697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1978.428697] alg: No test for adler32 (adler32-zlib) [ 1979.274275] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1979.357025] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1980.935164] Key type lgssc registered [ 1981.325647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1983.560258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1983.562456] LustreError: 110180:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1983.563039] Lustre: Skipped 1 previous similar message [ 1983.567831] LustreError: 110179:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1983.570326] LustreError: 110180:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1983.573969] LustreError: 110180:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1983.577539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.579331] LustreError: 110180:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1984.756899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.530786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1987.669867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1989.141922] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1990.971669] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing lsmod [ 1992.149090] LustreError: 111355:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1992.153180] LustreError: 111356:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1992.155859] LustreError: 111355:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1992.159143] LustreError: 111355:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1992.162896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1992.165660] LustreError: 111355:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1992.376961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.580375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.582984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1993.584762] Lustre: Skipped 2 previous similar messages [ 1993.587361] Lustre: Skipped 1 previous similar message [ 1997.731077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1997.733720] Lustre: Skipped 2 previous similar messages [ 1999.677568] LustreError: 111547:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1999.706770] Lustre: server umount lustre-MDT0000 complete [ 2001.158353] LustreError: 110193:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236860 with bad export cookie 15915598564048698223 [ 2001.162147] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2001.162747] LustreError: 110193:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2001.823259] LustreError: 110653:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000071a1b2d7 x1859373590658304/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2001.836986] LustreError: 111749:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2001.839913] LustreError: 111749:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2001.889272] Lustre: server umount lustre-MDT0001 complete [ 2013.490350] LustreError: 111952:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2013.493078] LustreError: 111952:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2013.537931] Lustre: server umount lustre-OST0000 complete [ 2016.601906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.772895] LustreError: 112428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.780945] LustreError: 112428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2016.798742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2018.236134] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2021.203183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2021.334961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2022.617776] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2024.544315] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2025.920053] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2028.466711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2030.626972] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2032.621421] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2041.901227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2046.946683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2046.947710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2046.951657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2046.954808] Lustre: Skipped 1 previous similar message [ 2046.958041] Lustre: Skipped 1 previous similar message [ 2050.797425] LustreError: 114697:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2050.800058] LustreError: 114697:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2050.854150] Lustre: server umount lustre-OST0000 complete [ 2057.184671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2057.184797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2057.185561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2057.189124] LustreError: Skipped 1 previous similar message [ 2057.193753] Lustre: Skipped 1 previous similar message [ 2058.430590] LustreError: 114899:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2058.433750] LustreError: 114899:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2058.464488] Lustre: server umount lustre-MDT0000 complete [ 2059.943523] LustreError: 112454:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236918 with bad export cookie 15915598564048698895 [ 2059.946743] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2059.947634] LustreError: 112454:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2060.093498] Lustre: server umount lustre-MDT0001 complete [ 2064.309790] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2065.516098] Key type lgssc unregistered [ 2065.690191] LNet: 115632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2065.699513] LNetError: 115632:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2065.715061] LNet: Removed LNI 192.168.204.142@tcp [ 2066.108993] Key type .llcrypt unregistered [ 2066.110471] Key type ._llcrypt unregistered [ 2075.750173] Key type ._llcrypt registered [ 2075.751657] Key type .llcrypt registered [ 2075.791482] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2076.165790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2076.176237] alg: No test for adler32 (adler32-zlib) [ 2077.052049] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 2077.158315] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2078.743183] Key type lgssc registered [ 2079.224675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2082.368378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2082.377089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2083.566051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2085.091798] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2088.602810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2088.758863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2090.379437] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2092.519237] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2093.845702] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2096.275947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2096.391611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.379567] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2100.196219] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2108.711102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2112.912219] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2113.683047] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2119.137600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2119.138521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.139654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2119.139941] LustreError: Skipped 1 previous similar message [ 2119.143613] Lustre: Skipped 1 previous similar message [ 2120.688027] LustreError: 119374:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2120.729470] Lustre: server umount lustre-OST0000 complete [ 2124.256544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.259038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.260829] Lustre: Skipped 1 previous similar message [ 2124.262680] Lustre: Skipped 1 previous similar message [ 2128.252286] LustreError: 119576:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2128.255228] LustreError: 119576:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2128.282462] Lustre: server umount lustre-MDT0000 complete [ 2129.376295] LustreError: 117722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2129.381021] LustreError: 117722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2129.578191] LustreError: 116986:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236988 with bad export cookie 13570174429870183603 [ 2129.580149] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2129.585029] LustreError: 116986:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2129.653630] LustreError: 119779:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2129.655989] LustreError: 119779:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2129.707973] Lustre: server umount lustre-MDT0001 complete [ 2133.307946] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2134.378180] Key type lgssc unregistered [ 2134.509436] LNet: 120311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2134.512114] LNetError: 120311:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2134.520369] LNet: Removed LNI 192.168.204.142@tcp [ 2134.800927] Key type .llcrypt unregistered [ 2134.802584] Key type ._llcrypt unregistered [ 2141.700347] Key type ._llcrypt registered [ 2141.701734] Key type .llcrypt registered [ 2141.745394] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2142.151589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2142.165244] alg: No test for adler32 (adler32-zlib) [ 2143.016621] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 2143.106620] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2144.687163] Key type lgssc registered [ 2145.054049] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2147.657588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2147.663940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.814738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2150.126975] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2152.892676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2153.007523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2154.177824] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2155.827496] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2157.112303] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2159.414398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2159.545106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2161.296288] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2163.054631] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2174.885651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2183.136230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2183.139987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.146198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2185.185433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2185.186554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.309080] LustreError: 123757:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2187.345400] Lustre: server umount lustre-OST0000 complete [ 2188.768090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2188.771272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.775549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2188.777665] Lustre: Skipped 1 previous similar message [ 2194.686890] LustreError: 123959:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2194.688930] LustreError: 123959:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2194.718362] Lustre: server umount lustre-MDT0000 complete [ 2195.423882] LustreError: 121490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.429665] LustreError: 121490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2195.950123] LustreError: 121472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237054 with bad export cookie 2377010376780991698 [ 2195.952564] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2195.954449] LustreError: 121472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2196.020371] LustreError: 124161:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2196.022442] LustreError: 124161:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2196.069299] Lustre: server umount lustre-MDT0001 complete [ 2199.263153] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2200.166042] Key type lgssc unregistered [ 2200.278274] LNet: 124691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2200.280388] LNetError: 124691:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2200.287293] LNet: Removed LNI 192.168.204.142@tcp [ 2200.576681] Key type .llcrypt unregistered [ 2200.578302] Key type ._llcrypt unregistered [ 2209.035044] Key type ._llcrypt registered [ 2209.036161] Key type .llcrypt registered [ 2209.075397] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2209.369171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2209.403560] alg: No test for adler32 (adler32-zlib) [ 2210.250718] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 2210.325895] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2211.895175] Key type lgssc registered [ 2212.211759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2214.600307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2214.605932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2215.783239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2217.011894] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2219.765592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2219.890856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2221.140648] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2222.963775] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2224.205502] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2226.438030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2226.556413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2228.446526] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2230.258470] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2240.936548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2243.346314] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2244.537648] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2244.539896] Lustre: Skipped 2 previous similar messages [ 2245.829053] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2245.831490] Lustre: Skipped 2 previous similar messages [ 2250.231830] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2250.241982] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2253.205792] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.42@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2253.209849] Lustre: Skipped 2 previous similar messages [ 2258.931933] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.42@tcp 192.168.0.[2,10]@tcp [ 2260.917183] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.42/24@tcp in log lustre-MDT0000 [ 2260.919396] Lustre: Skipped 2 previous similar messages [ 2268.153673] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.42/24@tcp [ 2268.156199] Lustre: Skipped 1 previous similar message [ 2270.688038] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2270.690245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2270.694992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2271.712580] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2271.712783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2271.714349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.720024] Lustre: Skipped 1 previous similar message [ 2272.045088] LustreError: 130073:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2272.079025] Lustre: server umount lustre-OST0000 complete [ 2275.389470] LustreError: 130274:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2275.393477] LustreError: 130274:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2275.422810] Lustre: server umount lustre-MDT0000 complete [ 2276.319863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2276.322267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2276.327962] LustreError: 126065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.333392] LustreError: 126065:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2276.670401] LustreError: 128386:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237135 with bad export cookie 15967583019230502005 [ 2276.673862] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2276.674292] LustreError: 128386:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2276.740508] LustreError: 130476:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2276.742889] LustreError: 130476:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2276.797574] Lustre: server umount lustre-MDT0001 complete [ 2280.238995] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2281.297722] Key type lgssc unregistered [ 2281.423363] LNet: 131007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.426461] LNetError: 131007:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2281.438300] LNet: Removed LNI 192.168.204.142@tcp [ 2281.720911] Key type .llcrypt unregistered [ 2281.722290] Key type ._llcrypt unregistered [ 2291.376572] Key type ._llcrypt registered [ 2291.377945] Key type .llcrypt registered [ 2291.415315] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2291.792684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2291.801425] alg: No test for adler32 (adler32-zlib) [ 2292.639434] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 2292.713929] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2294.287156] Key type lgssc registered [ 2294.604106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2296.972925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2296.980939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.120980] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2298.126399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.128289] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.42/24@tcp [ 2299.317179] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2301.916908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2302.011399] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2302.019166] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.42/24@tcp [ 2302.032487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2303.147671] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2304.779303] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2305.873185] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2307.953768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2308.059967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2309.797312] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2311.425585] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2312.180612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2322.402437] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2322.403150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2322.404838] LustreError: Skipped 1 previous similar message [ 2322.408731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2322.411309] Lustre: Skipped 1 previous similar message [ 2326.448118] LustreError: 134827:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2326.480911] Lustre: server umount lustre-OST0000 complete [ 2332.640801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.640827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2332.641777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2332.645357] Lustre: Skipped 2 previous similar messages [ 2333.821081] LustreError: 135028:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2333.823981] LustreError: 135028:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2333.856516] Lustre: server umount lustre-MDT0000 complete [ 2335.086479] LustreError: 132508:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237194 with bad export cookie 139278935055222937 [ 2335.089858] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2335.090539] LustreError: 132508:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2335.166970] LustreError: 135229:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2335.170129] LustreError: 135229:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2335.229253] Lustre: server umount lustre-MDT0001 complete [ 2338.795058] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2339.851455] Key type lgssc unregistered [ 2339.981426] LNet: 135760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2339.983930] LNetError: 135760:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2339.992379] LNet: Removed LNI 192.168.204.142@tcp [ 2340.333228] Key type .llcrypt unregistered [ 2340.334308] Key type ._llcrypt unregistered [ 2348.430192] Key type ._llcrypt registered [ 2348.431508] Key type .llcrypt registered [ 2348.464843] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 2354.396431] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2354.794325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2354.806328] alg: No test for adler32 (adler32-zlib) [ 2355.667077] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 2355.748465] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2357.327098] Key type lgssc registered [ 2357.663244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2360.380294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2362.496485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2364.458883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2366.496798] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2368.841078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2368.859425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2368.865420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.960974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2369.976588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2370.017475] Lustre: lustre-MDT0000: new disk, initializing [ 2370.043806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2370.050559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2371.414913] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2375.565262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2375.584597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2375.615888] Lustre: 139107:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2375.637124] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2375.638837] Lustre: Skipped 1 previous similar message [ 2375.676360] Lustre: lustre-MDT0001: new disk, initializing [ 2375.708113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2375.719301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2375.722125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2377.006235] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2380.008926] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2381.201477] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2383.552450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2383.579967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2383.712474] Lustre: lustre-OST0000: new disk, initializing [ 2383.714323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2383.742709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2385.493933] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2387.513663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2387.517269] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2387.552285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2388.482354] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2389.772636] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2389.849030] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2391.121258] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2391.195511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2392.547426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2392.548262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2392.549663] LustreError: Skipped 1 previous similar message [ 2392.554888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2392.558079] Lustre: Skipped 1 previous similar message [ 2393.838089] LustreError: 142043:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2393.876758] Lustre: server umount lustre-OST0000 complete [ 2396.127984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2396.130879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.135186] Lustre: Skipped 1 previous similar message [ 2396.137421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2397.664223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2397.664903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2401.277438] LustreError: 142243:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2401.279471] LustreError: 142243:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2401.304586] Lustre: server umount lustre-MDT0000 complete [ 2402.550399] LustreError: 139885:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237261 with bad export cookie 9945183015987175922 [ 2402.554993] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2402.555632] LustreError: 139885:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2402.621186] LustreError: 142445:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2402.623124] LustreError: 142445:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2402.681361] Lustre: server umount lustre-MDT0001 complete [ 2405.438052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.606277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2406.865522] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2409.467648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2410.788344] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2412.470135] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2413.550221] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2415.587136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2415.694893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2415.696475] Lustre: Skipped 1 previous similar message [ 2417.475235] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2419.125485] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2420.391808] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2420.469833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2421.670761] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2421.740847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2425.923210] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2425.948479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2426.004912] Lustre: lustre-OST0001: new disk, initializing [ 2426.006478] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2426.033494] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2426.193567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2426.197906] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2426.229036] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2427.737355] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2430.814170] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2430.885130] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2432.142893] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2432.214429] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2433.352858] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e3654f3c000.ost_server_uuid 50 [ 2433.810798] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e3654f3c000.ost_server_uuid in FULL state after 0 sec [ 2436.577817] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2436.579740] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2436.580084] LustreError: Skipped 1 previous similar message [ 2436.580865] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2436.580869] Lustre: Skipped 1 previous similar message [ 2436.583311] Lustre: Skipped 1 previous similar message [ 2441.516364] LustreError: 147165:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2441.518115] LustreError: 147165:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2441.534152] Lustre: server umount lustre-OST0001 complete [ 2441.699845] LustreError: 144516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.705031] LustreError: 144516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2445.792028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2445.794498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.799165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2445.800899] Lustre: Skipped 1 previous similar message [ 2446.818852] LustreError: 144708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.822792] LustreError: 144708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2451.938593] LustreError: 144516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.942808] LustreError: 144516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2457.055078] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2457.057366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2457.058818] LustreError: 147369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.064382] Lustre: Skipped 5 previous similar messages [ 2457.074832] LustreError: 147367:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2457.077083] LustreError: 147367:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2457.109226] Lustre: server umount lustre-OST0000 complete [ 2460.640045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2460.642384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2460.645830] Lustre: Skipped 1 previous similar message [ 2464.402250] Lustre: server umount lustre-MDT0000 complete [ 2465.569121] LustreError: 144537:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237324 with bad export cookie 9945183015987176797 [ 2465.571960] LustreError: 144537:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2465.572305] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2465.634623] LustreError: 147771:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2465.636548] LustreError: 147771:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2465.678188] Lustre: server umount lustre-MDT0001 complete [ 2471.500603] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2472.440403] Key type lgssc unregistered [ 2472.561373] LNet: 148758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2472.563993] LNetError: 148758:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2472.572326] LNet: Removed LNI 192.168.204.142@tcp [ 2472.853628] Key type .llcrypt unregistered [ 2472.854509] Key type ._llcrypt unregistered [ 2482.928853] Key type ._llcrypt registered [ 2482.929735] Key type .llcrypt registered [ 2482.962306] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 2488.607823] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2489.011762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2489.019180] alg: No test for adler32 (adler32-zlib) [ 2489.864648] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 2489.951325] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2491.527112] Key type lgssc registered [ 2491.869955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2494.501378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2496.437931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2498.305740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2500.165903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2502.246918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2502.265632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2502.269255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2503.374289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2503.387760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2503.432846] Lustre: lustre-MDT0000: new disk, initializing [ 2503.460189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2503.468534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2504.803430] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2508.977804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2509.002216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2509.032609] Lustre: 152679:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2509.054983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2509.057070] Lustre: Skipped 1 previous similar message [ 2509.095234] Lustre: lustre-MDT0001: new disk, initializing [ 2509.130238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2509.141130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2509.145679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2510.423911] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2513.444506] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2514.652665] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2516.921769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2516.945721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2517.047505] Lustre: lustre-OST0000: new disk, initializing [ 2517.049915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2517.075912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2518.071342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2518.074640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2518.110254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2518.964809] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2522.000536] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2523.320334] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2523.395429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2524.732575] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2524.820752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2527.533590] LustreError: 155618:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2527.574206] Lustre: server umount lustre-OST0000 complete [ 2529.759942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2529.762469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.766945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2533.344578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2533.344989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.464952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2538.467302] Lustre: Skipped 2 previous similar messages [ 2538.467938] LustreError: 155819:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2538.470851] LustreError: 155819:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2538.495711] Lustre: server umount lustre-MDT0000 complete [ 2539.702204] LustreError: 152669:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237398 with bad export cookie 141171972105321289 [ 2539.704921] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2539.706306] LustreError: 152669:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2549.219259] LustreError: 156021:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2549.221079] LustreError: 156021:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2549.277328] Lustre: server umount lustre-MDT0001 complete [ 2551.959802] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 09:56:51 (1773237411) [ 2557.169721] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 2560.192975] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2563.472323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2565.602268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2567.568820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2569.679984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2571.790148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2571.813045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.890867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2571.901803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2571.940424] Lustre: lustre-MDT0000: new disk, initializing [ 2571.961768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2571.967928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2573.183518] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2577.181164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2577.203018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2577.232236] Lustre: 159008:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2577.253454] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2577.255206] Lustre: Skipped 1 previous similar message [ 2577.288454] Lustre: lustre-MDT0001: new disk, initializing [ 2577.323973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2577.337220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2577.341434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2578.588652] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2581.520638] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2582.700663] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2584.886202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2584.908266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2586.888983] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2589.829127] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2590.195662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2590.199651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2590.226886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2597.034080] Lustre: Failing over lustre-OST0000 [ 2597.052065] LustreError: 161452:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2597.053948] LustreError: 161452:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2597.088994] Lustre: server umount lustre-OST0000 complete [ 2609.673052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2609.768572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.771690] Lustre: Skipped 1 previous similar message [ 2609.778415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.780789] Lustre: Skipped 1 previous similar message [ 2609.784826] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2611.445445] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2611.680465] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2612.607799] Lustre: Failing over lustre-MDT0000 [ 2612.672342] LustreError: 162425:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2612.674902] LustreError: 162425:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2612.701954] Lustre: server umount lustre-MDT0000 complete [ 2613.215710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2613.217948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2616.287152] Lustre: 150895:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773237459/real 1773237459] req@00000000c9b83cfe x1859374125473792/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1773237475 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2616.296651] Lustre: 150895:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2620.383949] LustreError: 159018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.388015] LustreError: 159018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2621.791135] Lustre: 150898:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1773237464/real 1773237464] req@000000008d0730bb x1859374125475712/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1773237480 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2621.911467] LustreError: 159013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.375996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.409084] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2625.413604] Lustre: 159010:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.204.142@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2625.477909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2625.514542] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2626.605927] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2627.036027] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2630.630408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2630.637100] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2630.639462] Lustre: Skipped 1 previous similar message [ 2630.653457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2635.746118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2635.748049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2635.750334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2635.751074] Lustre: Skipped 3 previous similar messages [ 2637.609814] LustreError: 163308:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2637.611699] LustreError: 163308:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2637.643405] Lustre: server umount lustre-OST0000 complete [ 2640.864995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2640.865782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2640.867765] LustreError: Skipped 1 previous similar message [ 2640.869735] Lustre: Skipped 1 previous similar message [ 2651.104116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2651.106010] Lustre: Skipped 4 previous similar messages [ 2653.151128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2653.209130] Lustre: server umount lustre-MDT0000 complete [ 2654.434319] LustreError: 158998:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237513 with bad export cookie 141171972105323340 [ 2654.437407] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2654.437689] LustreError: 158998:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2654.510318] LustreError: 163714:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2654.512360] LustreError: 163714:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2654.568570] Lustre: server umount lustre-MDT0001 complete [ 2658.305916] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2659.371445] Key type lgssc unregistered [ 2659.511427] LNet: 164245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2659.515044] LNetError: 164245:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2659.522347] LNet: Removed LNI 192.168.204.142@tcp [ 2659.838587] Key type .llcrypt unregistered [ 2659.840250] Key type ._llcrypt unregistered [ 2679.595642] Key type ._llcrypt registered [ 2679.596632] Key type .llcrypt registered [ 2679.634688] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2680.064843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2680.120519] alg: No test for adler32 (adler32-zlib) [ 2680.979368] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 2681.068379] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2682.655156] Key type lgssc registered [ 2683.029711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2685.626935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2685.633246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.788174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2688.014143] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2691.031324] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2693.018051] LustreError: 165770:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2693.021446] LustreError: 165770:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2693.024109] LustreError: 165770:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2693.086035] LustreError: 166292:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2693.149661] Lustre: server umount lustre-MDT0000 complete [ 2697.844556] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 2700.761833] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2703.874628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2705.924201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2707.833389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2709.931739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2712.144783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2712.164800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2712.246769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2712.256799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2712.294129] Lustre: lustre-MDT0000: new disk, initializing [ 2712.317539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.324193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2713.562730] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2717.569430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2717.591902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2717.622622] Lustre: 168929:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2717.647573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2717.651291] Lustre: Skipped 1 previous similar message [ 2717.696175] Lustre: lustre-MDT0001: new disk, initializing [ 2717.727015] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2717.742365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2717.746363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2718.994799] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2722.001697] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2723.138357] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2725.317244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2725.344561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2725.437551] Lustre: lustre-OST0000: new disk, initializing [ 2725.439316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2725.465894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.291387] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2730.343530] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2732.536395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2732.539211] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2732.569462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2737.220365] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2737.777222] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:59:57 (1773237597) [ 2804.191608] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2804.191765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2804.195258] Lustre: Skipped 2 previous similar messages [ 2804.197180] Lustre: Skipped 1 previous similar message [ 2808.895404] LustreError: 172443:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2808.898023] LustreError: 172443:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2808.934234] Lustre: server umount lustre-MDT0000 complete [ 2809.311760] LustreError: 168940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.318625] LustreError: 168940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2810.184535] LustreError: 169708:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237669 with bad export cookie 10878216065825008640 [ 2810.185799] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2810.190860] LustreError: 169708:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2814.431443] LustreError: 168936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2814.431541] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.441368] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2815.455357] LustreError: 168936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2815.455392] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2816.322539] LustreError: 172644:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2816.325378] LustreError: 172644:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2816.371681] Lustre: server umount lustre-MDT0001 complete [ 2823.722556] LustreError: 172845:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2823.724446] LustreError: 172845:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2823.758228] Lustre: server umount lustre-OST0000 complete [ 2826.592480] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 2829.303471] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2832.380630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2834.285505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2836.145872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2838.107979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2840.132046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2840.151571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2840.224906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2840.233892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2840.273760] Lustre: lustre-MDT0000: new disk, initializing [ 2840.295884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2840.300740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2841.428770] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2845.442443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2845.468208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2845.503147] Lustre: 175177:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2845.526250] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2845.528099] Lustre: Skipped 1 previous similar message [ 2845.564837] Lustre: lustre-MDT0001: new disk, initializing [ 2845.609852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2845.614058] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2846.913731] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2850.025042] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2851.163147] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2853.456571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2853.479780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2853.575372] Lustre: lustre-OST0000: new disk, initializing [ 2853.578467] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2853.607482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2853.610388] Lustre: Skipped 1 previous similar message [ 2855.423886] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2858.497727] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2859.841401] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2862.582392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2862.585531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2862.617609] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2862.979729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2864.256900] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2864.333710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2867.682321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2867.682717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2867.687727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2867.689615] LustreError: Skipped 1 previous similar message [ 2871.021106] LustreError: 178136:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2871.023198] LustreError: 178136:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2871.056935] Lustre: server umount lustre-OST0000 complete [ 2872.800267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2872.800855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2872.806692] Lustre: Skipped 2 previous similar messages [ 2872.808345] Lustre: Skipped 1 previous similar message [ 2878.424694] Lustre: server umount lustre-MDT0000 complete [ 2879.688207] LustreError: 175957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237738 with bad export cookie 10878216065825167456 [ 2879.690568] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2879.692180] LustreError: 175957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2879.758149] LustreError: 178539:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2879.759869] LustreError: 178539:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2879.814611] Lustre: server umount lustre-MDT0001 complete [ 2882.470341] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:02:21 (1773237741) [ 2887.623971] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 2890.713300] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2893.918194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2896.044904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2898.022121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2900.043186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2902.219380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2902.245746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.324076] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2902.334251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2902.370907] Lustre: lustre-MDT0000: new disk, initializing [ 2902.394155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2902.402156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2903.666092] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2907.750537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2907.776756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2907.812248] Lustre: 181523:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2907.917558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2909.239395] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2912.306754] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2913.508601] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2915.746043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2915.775895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2917.727315] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2918.518876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2918.550588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2920.746599] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2924.001951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2924.002124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2924.004271] LustreError: Skipped 1 previous similar message [ 2924.011159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2924.014259] Lustre: Skipped 4 previous similar messages [ 2928.619105] LustreError: 184070:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2928.621463] LustreError: 184070:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2928.653682] Lustre: server umount lustre-OST0000 complete [ 2933.727864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2933.730033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2933.732962] Lustre: Skipped 1 previous similar message [ 2936.090882] Lustre: server umount lustre-MDT0000 complete [ 2937.328268] LustreError: 182308:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237796 with bad export cookie 10878216065825168324 [ 2937.331056] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2937.334713] LustreError: 182308:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2937.459391] Lustre: server umount lustre-MDT0001 complete [ 2940.225499] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:03:19 (1773237799) [ 2945.028549] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 2947.686313] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2950.538207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2952.443903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2954.249574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2956.095061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2958.184031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2958.201156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2958.275408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2958.285339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2958.286664] Lustre: Skipped 3 previous similar messages [ 2958.318730] Lustre: lustre-MDT0000: new disk, initializing [ 2958.320028] Lustre: Skipped 2 previous similar messages [ 2958.340661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2958.342439] Lustre: Skipped 2 previous similar messages [ 2958.347341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2958.349317] Lustre: Skipped 2 previous similar messages [ 2959.511879] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2963.317291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2963.339296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2963.367595] Lustre: 187467:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2963.460245] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2964.579459] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2967.523622] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2968.682331] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2970.929629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2970.955893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2972.849344] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2973.028242] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2976.130930] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2978.273733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2978.274874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2978.277722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2978.277728] Lustre: Skipped 3 previous similar messages [ 2978.278197] LustreError: Skipped 1 previous similar message [ 2978.282339] Lustre: Skipped 2 previous similar messages [ 2984.046117] LustreError: 190010:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2984.048202] LustreError: 190010:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2984.087047] Lustre: server umount lustre-OST0000 complete [ 2991.451601] Lustre: server umount lustre-MDT0000 complete [ 2992.689218] LustreError: 187458:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237851 with bad export cookie 10878216065825169395 [ 2992.692852] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2992.693408] LustreError: 187458:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2992.815414] Lustre: server umount lustre-MDT0001 complete [ 2996.347646] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2997.428114] Key type lgssc unregistered [ 2997.561414] LNet: 190942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2997.564990] LNetError: 190942:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2997.575310] LNet: Removed LNI 192.168.204.142@tcp [ 2997.927269] Key type .llcrypt unregistered [ 2997.928920] Key type ._llcrypt unregistered [ 3006.923767] Key type ._llcrypt registered [ 3006.924879] Key type .llcrypt registered [ 3006.963038] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3007.355322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3007.419184] alg: No test for adler32 (adler32-zlib) [ 3008.291363] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3008.387476] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 3009.975238] Key type lgssc registered [ 3010.386029] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3012.922129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3012.928775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.079473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3015.360341] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3018.096406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3018.223102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3019.482110] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3021.193186] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3022.432428] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3024.692893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3024.835541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3026.746178] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3028.546331] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3039.207407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3059.681134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3059.682413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3059.683174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3059.683696] LustreError: Skipped 1 previous similar message [ 3059.687358] Lustre: Skipped 1 previous similar message [ 3061.999128] LustreError: 194590:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3062.039039] Lustre: server umount lustre-OST0000 complete [ 3064.287905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3064.290588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.295397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3064.297453] Lustre: Skipped 1 previous similar message [ 3069.436656] LustreError: 194793:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3069.438736] LustreError: 194793:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3069.466821] Lustre: server umount lustre-MDT0000 complete [ 3069.920185] LustreError: 192320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3069.926049] LustreError: 192320:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3070.707567] LustreError: 193936:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237929 with bad export cookie 787137588648574673 [ 3070.710797] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3070.713165] LustreError: 193936:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3070.787975] LustreError: 194995:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3070.790170] LustreError: 194995:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3070.843133] Lustre: server umount lustre-MDT0001 complete [ 3074.283723] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 3075.196966] Key type lgssc unregistered [ 3075.321369] LNet: 195526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3075.323441] LNetError: 195526:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3075.334255] LNet: Removed LNI 192.168.204.142@tcp [ 3075.591569] Key type .llcrypt unregistered [ 3075.592646] Key type ._llcrypt unregistered [ 3083.568231] Key type ._llcrypt registered [ 3083.569795] Key type .llcrypt registered [ 3083.604290] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3083.918550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3083.961370] alg: No test for adler32 (adler32-zlib) [ 3084.806036] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3084.885491] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 3086.463127] Key type lgssc registered [ 3086.775415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3089.135604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3089.141166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3090.273746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3091.417769] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3094.039774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3094.158848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3095.336721] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3096.987420] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3098.047245] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3100.116128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3100.221790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3101.285068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3101.933890] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3103.489076] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3106.783787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3106.785853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3106.789671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3109.854287] Lustre: lustre-OST0000: Not available for connect from 192.168.204.42@tcp (stopping) [ 3109.856780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3109.859598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3112.558041] LustreError: 199144:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3112.590422] Lustre: server umount lustre-OST0000 complete [ 3114.378158] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9e3643660000.ost_server_uuid 50 [ 3114.898180] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9e3643660000.ost_server_uuid in DISCONN state after 0 sec [ 3115.434796] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3129.568926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.569871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3129.572625] Lustre: Skipped 1 previous similar message [ 3129.575115] Lustre: Skipped 2 previous similar messages [ 3131.581314] LustreError: 199541:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3131.583190] LustreError: 199541:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3131.612671] Lustre: server umount lustre-MDT0000 complete [ 3132.882475] LustreError: 198518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237991 with bad export cookie 14419238340942395398 [ 3132.884646] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3132.887738] LustreError: 198518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3132.953916] LustreError: 199742:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3132.955675] LustreError: 199742:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3133.009769] Lustre: server umount lustre-MDT0001 complete [ 3135.869797] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:06:35 (1773237995) [ 3137.989139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3138.133238] LustreError: 200412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3138.138091] LustreError: 200412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3138.156232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3139.362292] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3142.009462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3143.308090] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3144.924180] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3146.069069] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3148.211371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3148.258061] LustreError: 202010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.312571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3148.314326] Lustre: Skipped 1 previous similar message [ 3149.985797] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3151.563984] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3152.421941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3153.701278] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3153.723168] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3153.771925] Lustre: lustre-OST0001: new disk, initializing [ 3153.773568] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3155.506546] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3158.421822] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3162.594597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3162.594967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3162.596743] LustreError: Skipped 1 previous similar message [ 3162.600602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3162.602640] Lustre: Skipped 2 previous similar messages [ 3163.634735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3163.638159] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3163.662703] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3165.293254] LustreError: 203605:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3165.295057] LustreError: 203605:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3165.310618] Lustre: server umount lustre-OST0000 complete [ 3166.992322] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3167.062248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3168.266350] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3168.337534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3168.729122] LustreError: 202008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.849399] LustreError: 202009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.855531] LustreError: 202009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3178.976868] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3178.977558] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3178.978162] LustreError: 202010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3178.978169] LustreError: 202010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3178.978699] LustreError: Skipped 1 previous similar message [ 3178.981321] Lustre: Skipped 1 previous similar message [ 3178.985518] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3178.991893] Lustre: Skipped 4 previous similar messages [ 3184.363429] LustreError: 204315:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3184.364946] LustreError: 204315:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3184.397762] Lustre: server umount lustre-OST0001 complete [ 3188.191779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3188.195710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.203481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3188.205445] Lustre: Skipped 2 previous similar messages [ 3191.706045] Lustre: server umount lustre-MDT0000 complete [ 3192.941372] LustreError: 200391:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238051 with bad export cookie 14419238340942396455 [ 3192.944503] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3192.948333] LustreError: 200391:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3193.015381] LustreError: 204720:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3193.018322] LustreError: 204720:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3193.073889] Lustre: server umount lustre-MDT0001 complete [ 3194.917117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3196.198220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3197.443620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3198.714838] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3200.934193] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:07:40 (1773238060) [ 3203.145868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3203.242756] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3203.254918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3203.323458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3203.325273] Lustre: Skipped 1 previous similar message [ 3204.630231] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3207.533518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3207.570632] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3207.584643] Lustre: 206209:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3209.022714] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3210.786625] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3211.968069] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3214.223255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3214.296873] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3216.300233] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3218.065311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3218.093496] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3220.392459] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3220.417404] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3220.512207] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3220.514116] Lustre: Skipped 2 previous similar messages [ 3222.477552] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3224.301041] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3225.569575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.571365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3225.572846] Lustre: Skipped 2 previous similar messages [ 3225.574633] Lustre: Skipped 2 previous similar messages [ 3239.391068] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3239.409086] LustreError: 209391:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3239.411292] LustreError: 209391:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3239.428669] Lustre: server umount lustre-OST0000 complete [ 3239.897193] LustreError: 207835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3239.904538] LustreError: 207835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3252.192501] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3256.147142] Lustre: server umount lustre-OST0001 complete [ 3258.847909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3258.854518] Lustre: Skipped 2 previous similar messages [ 3258.857342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3258.860448] Lustre: Skipped 10 previous similar messages [ 3263.581134] Lustre: server umount lustre-MDT0000 complete [ 3264.819151] LustreError: 206201:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238123 with bad export cookie 14419238340942397813 [ 3264.822846] LustreError: 206201:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3264.823965] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3264.928655] Lustre: server umount lustre-MDT0001 complete [ 3266.836775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3268.138516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3269.416171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3270.661984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3272.785525] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:08:52 (1773238132) [ 3277.894636] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 3280.808817] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3284.063236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3286.191711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3288.230654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3290.346107] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3292.681333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3292.706608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3292.786548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3292.799673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3292.838058] Lustre: lustre-MDT0000: new disk, initializing [ 3292.865259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3292.872640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3294.169770] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3298.325962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3298.349590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3298.382700] Lustre: 213827:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3298.412320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3298.415260] Lustre: Skipped 1 previous similar message [ 3298.457163] Lustre: lustre-MDT0001: new disk, initializing [ 3298.502463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3298.507397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3299.851549] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3303.013609] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3304.290791] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3306.442616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3306.461121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3306.547128] Lustre: lustre-OST0000: new disk, initializing [ 3306.549504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3308.292457] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3311.168196] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3312.117052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3312.121370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3312.153450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3312.361404] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3312.428086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3313.582251] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3313.648575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3317.217828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3317.219853] LustreError: Skipped 3 previous similar messages [ 3320.300544] LustreError: 216764:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3320.302305] LustreError: 216764:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3320.336813] Lustre: server umount lustre-OST0000 complete [ 3327.455800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3327.458644] Lustre: Skipped 8 previous similar messages [ 3327.635132] Lustre: server umount lustre-MDT0000 complete [ 3328.788290] LustreError: 213818:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238187 with bad export cookie 14419238340942399332 [ 3328.791268] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3328.792346] LustreError: 213818:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3328.910229] Lustre: server umount lustre-MDT0001 complete [ 3331.681755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3331.829138] LustreError: 217642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3331.836639] LustreError: 217642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3333.101276] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3335.825957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3337.153082] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3338.788569] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3339.884843] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3341.961421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3343.750690] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3345.353889] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3346.565334] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3356.837323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3358.135759] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3358.223804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3360.736211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3360.738893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3360.742541] Lustre: Skipped 5 previous similar messages [ 3364.942111] Lustre: server umount lustre-OST0000 complete [ 3367.035794] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3368.463181] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3392.074383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3392.184816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.187668] Lustre: Skipped 5 previous similar messages [ 3394.004601] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3395.731133] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3396.268608] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3399.276146] LustreError: 222085:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3399.278781] LustreError: 222085:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3399.319375] Lustre: server umount lustre-OST0000 complete [ 3402.714288] Lustre: server umount lustre-MDT0000 complete [ 3402.722608] LustreError: 217666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.727051] LustreError: 217666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3403.977163] LustreError: 217623:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238262 with bad export cookie 14419238340942400214 [ 3403.980185] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3403.983250] LustreError: 217623:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3404.106969] Lustre: server umount lustre-MDT0001 complete [ 3406.886978] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:11:06 (1773238266) [ 3409.045540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.468735] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3413.270476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3414.665670] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3416.482925] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3417.727396] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3419.901175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3421.828706] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3423.559265] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3424.918087] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3435.209775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3436.499458] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3436.580351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3438.750725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3438.775779] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3438.828399] Lustre: lustre-OST0001: new disk, initializing [ 3438.830127] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3440.697531] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3443.728889] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3445.041993] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3448.328079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3448.331493] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3448.371437] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3449.209339] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3450.496204] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3450.569595] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3453.410275] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3453.413731] LustreError: Skipped 3 previous similar messages [ 3457.275960] Lustre: server umount lustre-OST0001 complete [ 3459.180784] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3459.252126] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3460.529315] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3460.606589] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3468.772361] LustreError: 228072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3468.779955] LustreError: 228072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 3484.138022] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3486.081507] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3487.848022] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3488.348188] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3489.255193] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3489.257880] Lustre: Skipped 6 previous similar messages [ 3495.101577] Lustre: server umount lustre-OST0001 complete [ 3509.730627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.736257] Lustre: Skipped 5 previous similar messages [ 3513.311182] Lustre: server umount lustre-OST0000 complete [ 3520.862743] Lustre: server umount lustre-MDT0000 complete [ 3522.099626] LustreError: 223141:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238381 with bad export cookie 14419238340942401138 [ 3522.103947] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3528.258470] LustreError: 229740:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3528.260492] LustreError: 229740:0:(obd_class.h:479:obd_check_dev()) Skipped 37 previous similar messages [ 3528.320604] Lustre: server umount lustre-MDT0001 complete [ 3530.183434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3531.514478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3532.839359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3534.238925] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3536.422328] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:13:15 (1773238395) [ 3538.644139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3538.748852] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3538.761660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3538.840828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3538.842813] Lustre: Skipped 5 previous similar messages [ 3540.097988] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3543.008574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3543.043138] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3543.053142] Lustre: 231232:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3544.454517] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3546.256746] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3547.415387] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3549.687666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3549.766649] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3551.842854] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3553.761179] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3557.641476] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3559.651697] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3561.464795] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3562.009033] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3563.366283] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3563.455422] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3564.720830] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3564.796193] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3565.947659] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e3651039000.ost_server_uuid 50 [ 3566.424697] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e3651039000.ost_server_uuid in FULL state after 0 sec [ 3566.757208] Lustre: Permanently deactivating lustre-OST0001 [ 3566.762678] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3567.546029] Lustre: Permanently reactivating lustre-OST0001 [ 3574.595202] Lustre: server umount lustre-OST0001 complete [ 3590.111147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3590.163611] Lustre: server umount lustre-OST0000 complete [ 3597.542606] Lustre: server umount lustre-MDT0000 complete [ 3598.782331] LustreError: 231957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238457 with bad export cookie 14419238340942402405 [ 3598.785096] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3598.785505] LustreError: 231957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3598.917581] Lustre: server umount lustre-MDT0001 complete [ 3600.855421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3602.208886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3603.446948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3604.756986] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3607.007725] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:14:26 (1773238466) [ 3607.350245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3609.670784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3609.783498] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3609.794099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3609.798048] Lustre: Skipped 5 previous similar messages [ 3611.213901] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3613.997613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3614.029769] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3614.031792] Lustre: Skipped 1 previous similar message [ 3614.041841] Lustre: 237372:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3615.415696] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3617.168776] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3618.301147] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3620.636734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3620.661887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3620.740392] Lustre: Permanently deactivating lustre-OST0000 [ 3620.745427] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3622.673094] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3624.458145] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3626.757257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3626.959159] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3626.962330] LustreError: 239947:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -108 [ 3627.000487] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3628.727610] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3630.583357] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3631.205168] Lustre: Permanently reactivating lustre-OST0000 [ 3637.245281] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3637.248204] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3637.252609] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3637.266510] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3637.273258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3642.337124] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3642.340761] LustreError: Skipped 2 previous similar messages [ 3647.297896] Lustre: server umount lustre-OST0001 complete [ 3647.463082] LustreError: 239951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3647.469209] LustreError: 239951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [ 3654.678354] Lustre: server umount lustre-OST0000 complete [ 3662.110463] Lustre: server umount lustre-MDT0000 complete [ 3663.338277] LustreError: 237363:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238522 with bad export cookie 14419238340942404043 [ 3663.341133] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3663.342321] LustreError: 237363:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3663.467740] Lustre: server umount lustre-MDT0001 complete [ 3667.130158] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 3668.178378] Key type lgssc unregistered [ 3668.310356] LNet: 241761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3668.313735] LNetError: 241761:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3668.320319] LNet: Removed LNI 192.168.204.142@tcp [ 3668.621480] Key type .llcrypt unregistered [ 3668.622937] Key type ._llcrypt unregistered [ 3677.127016] Key type ._llcrypt registered [ 3677.128826] Key type .llcrypt registered [ 3677.167791] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3677.620324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3677.638224] alg: No test for adler32 (adler32-zlib) [ 3678.485154] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3678.569217] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 3680.151115] Key type lgssc registered [ 3680.510534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3681.661043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3683.848401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3683.856559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3685.032373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3686.302681] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3689.196180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3689.225345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3689.257241] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3689.269255] Lustre: 243174:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3689.273660] Lustre: Permanently deactivating lustre-MDT0001 [ 3689.279178] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3689.395499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3690.707550] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3691.627985] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3691.630454] LustreError: 243216:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3691.634594] LustreError: 243216:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3691.637775] LustreError: 243216:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 3692.519098] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3693.766899] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3695.985682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3696.114612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3698.109758] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3699.175686] LustreError: 244805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3699.178881] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3699.181057] LustreError: 244805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3699.858026] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3702.150378] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3702.220620] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3704.099223] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3705.010572] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3705.822167] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3710.120353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3710.121593] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3712.888724] Lustre: Permanently reactivating lustre-MDT0001 [ 3712.894056] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3712.896012] Lustre: Skipped 2 previous similar messages [ 3712.928686] Lustre: lustre-MDT0001: Client 3b452187-8467-4d98-8076-12bb695b5030 (at 192.168.204.42@tcp) reconnecting [ 3714.027150] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3714.565361] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3716.243815] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3716.312242] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3717.654656] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3718.760507] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 3719.148453] Lustre: 243181:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 3719.152245] Lustre: 243181:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 3719.154749] Lustre: 243181:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 3719.157372] Lustre: 243181:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 3719.160065] Lustre: 243181:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 3719.162573] Lustre: 243181:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 3719.584675] Lustre: Permanently deactivating lustre-MDT0001 [ 3719.590024] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3719.592415] Lustre: Skipped 2 previous similar messages [ 3727.361911] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3732.635401] Lustre: 243182:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 504 < left 979, rollback = 2 [ 3732.638255] Lustre: 243182:0:(osd_internal.h:1471:osd_trans_exec_op()) Skipped 1 previous similar message [ 3732.640520] Lustre: 243182:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/8, destroy: 0/0/0 [ 3732.642880] Lustre: 243182:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3732.645294] Lustre: 243182:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 6/979/0 [ 3732.647987] Lustre: 243182:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3732.650435] Lustre: 243182:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 3732.652963] Lustre: 243182:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3732.655524] Lustre: 243182:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 8/130/4, delete: 0/0/0 [ 3732.657893] Lustre: 243182:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3732.660456] Lustre: 243182:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0 [ 3732.663078] Lustre: 243182:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 3735.519702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3735.522048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3735.526221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3736.034692] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3736.037794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3736.039768] Lustre: Skipped 2 previous similar messages [ 3736.041725] Lustre: Skipped 2 previous similar messages [ 3739.842477] LustreError: 247354:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3739.882675] Lustre: server umount lustre-MDT0000 complete [ 3741.121575] LustreError: 243165:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238600 with bad export cookie 4642180863789037516 [ 3741.126870] LustreError: 243165:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3741.127190] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3741.151740] LustreError: 243180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3741.157144] LustreError: 243180:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3741.200210] LustreError: 247556:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3741.203408] LustreError: 247556:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3741.256599] Lustre: server umount lustre-MDT0001 complete [ 3748.526133] LustreError: 247757:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3748.528269] LustreError: 247757:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3748.542482] Lustre: server umount lustre-OST0000 complete [ 3752.182207] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:16:51 (1773238611) [ 3755.650301] LustreError: 248761:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3755.652413] LustreError: 248761:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3755.688363] Lustre: server umount lustre-OST0001 complete [ 3757.695439] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 3760.721653] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3763.865458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3765.972689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3768.025782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3770.122233] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3772.449514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3772.475618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3772.567843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3772.570804] Lustre: Skipped 2 previous similar messages [ 3772.579940] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3772.624306] Lustre: lustre-MDT0000: new disk, initializing [ 3772.648801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3772.655341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3774.048897] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3778.211855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3778.231898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3778.261056] Lustre: 250941:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3778.282931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3778.285263] Lustre: Skipped 1 previous similar message [ 3778.324084] Lustre: lustre-MDT0001: new disk, initializing [ 3778.366835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3778.371089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3779.608727] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3782.746798] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3784.001922] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3786.330460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3786.351433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3786.463502] Lustre: lustre-OST0000: new disk, initializing [ 3786.465450] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3786.504976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3786.508298] Lustre: Skipped 1 previous similar message [ 3788.412129] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3791.354451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3791.357634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3791.391902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3791.474018] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3792.087592] LustreError: 251730:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3796.242308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3796.266718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3796.325873] Lustre: lustre-OST0001: new disk, initializing [ 3796.327956] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3798.285541] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3801.383693] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3801.598701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3801.601757] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3801.632787] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3801.900391] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3802.183079] LustreError: 251730:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3802.192515] LustreError: 251730:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3812.287071] LustreError: 251730:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3815.903916] Lustre: lustre-OST0001: Not available for connect from 192.168.204.42@tcp (stopping) [ 3816.932268] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3816.933077] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3819.182074] LustreError: 254462:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3819.185103] LustreError: 254462:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3819.208085] Lustre: server umount lustre-OST0001 complete [ 3822.055508] LustreError: 252670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3822.060846] LustreError: 252670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3827.175016] LustreError: 253266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3827.183609] LustreError: 253266:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3832.293916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.294159] LustreError: 253266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3832.295087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3832.295094] LustreError: Skipped 1 previous similar message [ 3832.299102] Lustre: Skipped 1 previous similar message [ 3832.303341] LustreError: 253266:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3832.313750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3832.316484] Lustre: Skipped 3 previous similar messages [ 3837.294095] LustreError: 254685:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3837.296031] LustreError: 254685:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3837.341810] Lustre: server umount lustre-OST0000 complete [ 3839.968142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3839.970819] LustreError: Skipped 1 previous similar message [ 3839.972335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3839.976929] Lustre: Skipped 1 previous similar message [ 3839.979583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3844.865842] Lustre: server umount lustre-MDT0000 complete [ 3846.130334] LustreError: 250933:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238705 with bad export cookie 4642180863789040330 [ 3846.134587] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3846.135576] LustreError: 250933:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3852.352768] Lustre: server umount lustre-MDT0001 complete [ 3855.901956] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 3856.941780] Key type lgssc unregistered [ 3857.077547] LNet: 255618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3857.079810] LNetError: 255618:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3857.091399] LNet: Removed LNI 192.168.204.142@tcp [ 3857.425073] Key type .llcrypt unregistered [ 3857.426165] Key type ._llcrypt unregistered [ 3868.252338] Key type ._llcrypt registered [ 3868.253390] Key type .llcrypt registered [ 3868.291897] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 3874.353830] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3874.771716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3874.786236] alg: No test for adler32 (adler32-zlib) [ 3875.632334] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3875.715522] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 3877.295151] Key type lgssc registered [ 3877.659093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3880.601481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3882.756771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3884.757722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3886.879661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3889.104707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3889.123445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3889.127713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3890.208594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3890.220457] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3890.258587] Lustre: lustre-MDT0000: new disk, initializing [ 3890.282426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3890.289425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3891.548127] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3895.698343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3895.720838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3895.760855] Lustre: 259539:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3895.787286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3895.789412] Lustre: Skipped 1 previous similar message [ 3895.830561] Lustre: lustre-MDT0001: new disk, initializing [ 3895.868749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3895.885048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3895.889458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3897.224943] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3900.252255] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3901.488736] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3903.772819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3903.800469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3903.920918] Lustre: lustre-OST0000: new disk, initializing [ 3903.923543] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3903.960720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3905.837298] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3908.926655] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3910.240186] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3910.661439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3910.668961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3910.711218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3911.340032] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3912.576147] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3912.648029] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3915.744348] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3915.746047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3915.747960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3915.752014] Lustre: Skipped 1 previous similar message [ 3919.340921] LustreError: 262484:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3919.382321] Lustre: server umount lustre-OST0000 complete [ 3920.864953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3920.865509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3920.870234] Lustre: Skipped 1 previous similar message [ 3920.872448] Lustre: Skipped 1 previous similar message [ 3925.984345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3925.986281] Lustre: Skipped 2 previous similar messages [ 3926.781570] LustreError: 262686:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3926.783504] LustreError: 262686:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3926.811902] Lustre: server umount lustre-MDT0000 complete [ 3928.004132] LustreError: 259532:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238787 with bad export cookie 16635259893999505224 [ 3928.007347] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3928.008867] LustreError: 259532:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3928.074306] LustreError: 262888:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3928.076140] LustreError: 262888:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3928.126391] Lustre: server umount lustre-MDT0001 complete [ 3930.945695] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 10:19:50 (1773238790) [ 3933.122121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3933.305491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3934.572829] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3937.329564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3937.485405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3938.749341] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3940.688579] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3942.290280] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3944.688095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3946.773819] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3948.611484] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3999.202543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.203420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3999.207029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3999.210246] LustreError: Skipped 1 previous similar message [ 4003.375642] LustreError: 265904:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4003.378202] LustreError: 265904:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4003.435857] Lustre: server umount lustre-OST0000 complete [ 4004.888343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4009.313911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4009.444242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4009.447628] Lustre: Skipped 1 previous similar message [ 4011.290401] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4013.041977] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4019.685100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4019.688887] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4019.688899] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4050.402641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4050.403431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4050.404275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4050.404279] Lustre: Skipped 1 previous similar message [ 4050.406249] Lustre: Skipped 2 previous similar messages [ 4050.408535] LustreError: Skipped 2 previous similar messages [ 4052.524062] LustreError: 267567:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4052.526437] LustreError: 267567:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4052.572834] Lustre: server umount lustre-OST0000 complete [ 4055.521486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4055.522621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4055.524473] Lustre: Skipped 1 previous similar message [ 4060.640045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4060.641827] Lustre: Skipped 4 previous similar messages [ 4068.319103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4068.368121] LustreError: 267768:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4068.372743] LustreError: 267768:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4068.425556] Lustre: server umount lustre-MDT0000 complete [ 4069.768487] LustreError: 263538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238928 with bad export cookie 16635259893999506078 [ 4069.772791] LustreError: 263538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4069.773195] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4069.899827] Lustre: server umount lustre-MDT0001 complete [ 4073.333532] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 4074.408205] Key type lgssc unregistered [ 4074.546467] LNet: 268502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4074.549048] LNetError: 268502:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4074.555441] LNet: Removed LNI 192.168.204.142@tcp [ 4074.871101] Key type .llcrypt unregistered [ 4074.872192] Key type ._llcrypt unregistered [ 4083.502061] Key type ._llcrypt registered [ 4083.503378] Key type .llcrypt registered [ 4083.538426] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4083.938722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4083.952396] alg: No test for adler32 (adler32-zlib) [ 4084.800484] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4084.878091] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 4086.455166] Key type lgssc registered [ 4086.825798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4089.339534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4089.347802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4090.513626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4091.722314] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4094.561141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4094.682831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4095.935800] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4097.646118] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4098.876556] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4101.146423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4101.291507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4103.228682] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4104.977993] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4116.651038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4130.271956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4130.274247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4130.277917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4132.321335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4132.323191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4132.324657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4134.126113] LustreError: 272917:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4134.164910] Lustre: server umount lustre-OST0000 complete [ 4135.904071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4135.906795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4135.912225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4135.914158] Lustre: Skipped 1 previous similar message [ 4141.501582] LustreError: 273121:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4141.503975] LustreError: 273121:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4141.529381] Lustre: server umount lustre-MDT0000 complete [ 4142.559948] LustreError: 269877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4142.565353] LustreError: 269877:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4142.723043] LustreError: 269857:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239001 with bad export cookie 6999687422177520089 [ 4142.726276] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4142.726839] LustreError: 269857:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4142.802074] LustreError: 273324:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4142.803858] LustreError: 273324:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4142.851868] Lustre: server umount lustre-MDT0001 complete [ 4146.665774] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 4147.715871] Key type lgssc unregistered [ 4147.849380] LNet: 273855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4147.852616] LNetError: 273855:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4147.860320] LNet: Removed LNI 192.168.204.142@tcp [ 4148.139229] Key type .llcrypt unregistered [ 4148.140696] Key type ._llcrypt unregistered [ 4154.877848] Key type ._llcrypt registered [ 4154.878873] Key type .llcrypt registered [ 4154.913614] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4155.245424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4155.292996] alg: No test for adler32 (adler32-zlib) [ 4156.146661] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4156.224161] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 4157.799133] Key type lgssc registered [ 4158.135972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4160.784339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4160.791129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4161.965075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4163.333453] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4167.010988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4167.163299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4168.461067] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4170.405168] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4171.829793] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4174.187194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4174.302681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4176.046225] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4177.744026] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4186.599390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4191.716800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4191.718453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4191.718814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4191.722448] Lustre: Skipped 1 previous similar message [ 4193.581341] LustreError: 277448:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4193.613778] Lustre: server umount lustre-OST0000 complete [ 4196.832651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.833491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4196.836471] Lustre: Skipped 1 previous similar message [ 4196.838166] Lustre: Skipped 1 previous similar message [ 4197.050880] LustreError: 277648:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4197.052657] LustreError: 277648:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4197.079059] Lustre: server umount lustre-MDT0000 complete [ 4198.261139] LustreError: 275015:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239057 with bad export cookie 13815464200800111231 [ 4198.263840] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4198.264348] LustreError: 275015:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4198.335101] LustreError: 277849:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4198.337135] LustreError: 277849:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4198.384919] Lustre: server umount lustre-MDT0001 complete [ 4202.182074] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 4203.137910] Key type lgssc unregistered [ 4203.262334] LNet: 278380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4203.264439] LNetError: 278380:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4203.272314] LNet: Removed LNI 192.168.204.142@tcp [ 4203.565336] Key type .llcrypt unregistered [ 4203.566535] Key type ._llcrypt unregistered [ 4212.574940] Key type ._llcrypt registered [ 4212.576556] Key type .llcrypt registered [ 4212.610579] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4212.903797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4212.936069] alg: No test for adler32 (adler32-zlib) [ 4213.792062] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4213.877315] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 4215.455150] Key type lgssc registered [ 4215.825780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4218.596092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4218.602067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4219.764512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4220.952374] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4224.085288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4224.219212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4225.580152] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4227.296696] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4228.392436] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4230.745642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4230.869816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4232.695147] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4234.350053] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4244.203099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4254.689377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4254.690901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4254.691828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4254.697245] Lustre: Skipped 1 previous similar message [ 4254.700736] Lustre: Skipped 1 previous similar message [ 4258.031167] LustreError: 282796:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4258.081288] Lustre: server umount lustre-OST0000 complete [ 4259.808974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.809830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4259.812292] Lustre: Skipped 1 previous similar message [ 4264.928903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4264.931845] Lustre: Skipped 2 previous similar messages [ 4265.479445] LustreError: 282997:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4265.481387] LustreError: 282997:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4265.509096] Lustre: server umount lustre-MDT0000 complete [ 4266.771757] LustreError: 280474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239125 with bad export cookie 8152970564968054819 [ 4266.776992] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4266.778252] LustreError: 280474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4266.850359] LustreError: 283200:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4266.852184] LustreError: 283200:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4266.905868] Lustre: server umount lustre-MDT0001 complete [ 4270.284653] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 4271.264097] Key type lgssc unregistered [ 4271.389341] LNet: 283731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4271.391822] LNetError: 283731:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4271.403263] LNet: Removed LNI 192.168.204.142@tcp [ 4271.705125] Key type .llcrypt unregistered [ 4271.706223] Key type ._llcrypt unregistered [ 4278.507227] Key type ._llcrypt registered [ 4278.509019] Key type .llcrypt registered [ 4278.566176] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4279.035577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4279.071483] alg: No test for adler32 (adler32-zlib) [ 4279.952419] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4280.045037] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 4281.631190] Key type lgssc registered [ 4281.981282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4284.814360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4284.820926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4285.984277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4287.333097] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4290.190469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4290.313595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4291.566248] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4293.327572] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4294.629365] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4296.866797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4296.990456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4298.869403] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4300.640805] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4310.315289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4315.618821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4315.621193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4315.622271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4315.624607] Lustre: Skipped 1 previous similar message [ 4315.629321] Lustre: Skipped 1 previous similar message [ 4316.526075] LustreError: 287358:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4316.562764] Lustre: server umount lustre-OST0000 complete [ 4319.946898] LustreError: 287560:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4319.950952] LustreError: 287560:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4319.984469] Lustre: server umount lustre-MDT0000 complete [ 4320.736418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4320.740850] LustreError: 284937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4320.746435] LustreError: 284937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4321.193394] LustreError: 284975:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239180 with bad export cookie 7656317488035794971 [ 4321.196687] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4321.197449] LustreError: 284975:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4321.274101] LustreError: 287761:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4321.275890] LustreError: 287761:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4321.330583] Lustre: server umount lustre-MDT0001 complete [ 4325.077118] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 4326.142035] Key type lgssc unregistered [ 4326.265312] LNet: 288292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4326.267801] LNetError: 288292:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4326.274347] LNet: Removed LNI 192.168.204.142@tcp [ 4326.575162] Key type .llcrypt unregistered [ 4326.576305] Key type ._llcrypt unregistered [ 4330.105786] Key type ._llcrypt registered [ 4330.107447] Key type .llcrypt registered [ 4330.146508] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4335.669512] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 4341.330990] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4341.607649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4341.618052] alg: No test for adler32 (adler32-zlib) [ 4342.461641] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4342.536256] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 4344.111151] Key type lgssc registered [ 4344.465139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4347.176602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4349.179645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4351.156644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4353.149318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4355.502458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4355.523613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4355.529556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4356.617650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4356.630358] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4356.674068] Lustre: lustre-MDT0000: new disk, initializing [ 4356.701271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4356.708957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4357.927767] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4362.352263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4362.379671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4362.412449] Lustre: 291774:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4362.438187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4362.439848] Lustre: Skipped 1 previous similar message [ 4362.474638] Lustre: lustre-MDT0001: new disk, initializing [ 4362.512205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4362.524638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4362.528146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4363.989493] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4367.059971] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4368.196858] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4370.361497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4370.383606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4370.475711] Lustre: lustre-OST0000: new disk, initializing [ 4370.477750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4370.507317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4372.310216] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4374.331513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4374.335012] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4374.372603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4375.387059] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4376.701416] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4376.774490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4378.076597] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4378.167023] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4379.618347] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4379.620321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4379.622058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4379.625399] Lustre: Skipped 1 previous similar message [ 4380.848079] LustreError: 294714:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4380.893018] Lustre: server umount lustre-OST0000 complete [ 4383.199908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4383.202129] LustreError: Skipped 1 previous similar message [ 4383.203376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.207635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4383.209357] Lustre: Skipped 1 previous similar message [ 4384.736203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4384.736936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4388.286633] LustreError: 294914:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4388.289447] LustreError: 294914:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4388.318607] Lustre: server umount lustre-MDT0000 complete [ 4389.559742] LustreError: 291764:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239248 with bad export cookie 11383761799790527659 [ 4389.564731] LustreError: 291764:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4389.565182] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4389.632027] LustreError: 295116:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4389.633828] LustreError: 295116:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4389.690525] Lustre: server umount lustre-MDT0001 complete [ 4392.443413] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:27:31 (1773239251) [ 4394.711541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4394.897115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4396.178032] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4399.029840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4399.145538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4400.397754] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4402.106174] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4403.194685] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4405.401689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4407.434734] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4409.166242] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4430.303862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4430.306703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4430.311297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4430.313071] Lustre: Skipped 1 previous similar message [ 4433.517130] LustreError: 298056:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4433.519292] LustreError: 298056:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4433.558520] Lustre: server umount lustre-OST0000 complete [ 4434.911795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4434.914075] LustreError: Skipped 1 previous similar message [ 4434.915425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4434.919008] Lustre: Skipped 1 previous similar message [ 4434.920581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4434.922320] Lustre: Skipped 2 previous similar messages [ 4446.688618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4446.691173] Lustre: Skipped 5 previous similar messages [ 4449.247072] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4449.281604] LustreError: 298258:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4449.283576] LustreError: 298258:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4449.310062] Lustre: server umount lustre-MDT0000 complete [ 4450.519082] LustreError: 295766:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239309 with bad export cookie 11383761799790528527 [ 4450.522057] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4450.522320] LustreError: 295766:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4450.638631] Lustre: server umount lustre-MDT0001 complete [ 4454.055846] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 4455.198708] Key type lgssc unregistered [ 4455.334430] LNet: 298992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4455.336992] LNetError: 298992:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4455.347486] LNet: Removed LNI 192.168.204.142@tcp [ 4455.671772] Key type .llcrypt unregistered [ 4455.673705] Key type ._llcrypt unregistered [ 4464.049468] Key type ._llcrypt registered [ 4464.050598] Key type .llcrypt registered [ 4464.085763] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 4469.922635] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4470.187308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4470.237070] alg: No test for adler32 (adler32-zlib) [ 4471.080860] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4471.162268] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 4472.743164] Key type lgssc registered [ 4473.116968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4476.017937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4478.071185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4480.113469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4482.280629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4484.015438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4485.686104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4487.905167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4487.929462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4487.934118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4489.019433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4489.030082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4489.065849] Lustre: lustre-MDT0000: new disk, initializing [ 4489.093733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4489.100728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4490.322572] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4494.403842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4494.429218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4494.460602] Lustre: 302767:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4494.485443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4494.487143] Lustre: Skipped 1 previous similar message [ 4494.522546] Lustre: lustre-MDT0001: new disk, initializing [ 4494.555077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4494.567267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4494.570564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4495.769492] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4498.818907] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4500.018179] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4502.195358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4502.217377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4502.305455] Lustre: lustre-OST2710: new disk, initializing [ 4502.307579] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4502.335924] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4504.056899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4504.059941] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4504.062300] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4504.090567] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4507.018907] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4509.332879] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4509.359667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4509.423315] Lustre: lustre-OST03e8: new disk, initializing [ 4509.425060] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4509.457498] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4511.381957] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4513.104234] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4513.217027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4513.222128] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4513.250125] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4516.751567] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4516.828702] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4518.157324] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4518.246185] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4519.557774] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4519.644043] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4520.957462] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4521.042413] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4523.493626] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4523.496034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4523.497811] Lustre: Skipped 3 previous similar messages [ 4523.499300] Lustre: Skipped 2 previous similar messages [ 4528.189661] LustreError: 307107:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4528.219969] Lustre: server umount lustre-MDT0000 complete [ 4529.442305] LustreError: 302759:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239388 with bad export cookie 13535062284315476481 [ 4529.447850] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4529.448499] LustreError: 302759:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4529.518099] LustreError: 307308:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4529.519917] LustreError: 307308:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4529.572699] Lustre: server umount lustre-MDT0001 complete [ 4540.916113] LustreError: 307509:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4540.917980] LustreError: 307509:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4540.939955] Lustre: server umount lustre-OST2710 complete [ 4552.435807] LustreError: 307711:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4552.437805] LustreError: 307711:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4552.484805] Lustre: server umount lustre-OST03e8 complete [ 4558.261839] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 4561.364191] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4564.363408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4566.557543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4568.506475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4570.646841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4573.071052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4573.096777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4573.198372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4573.209942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4573.248153] Lustre: lustre-MDT0000: new disk, initializing [ 4573.275518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4573.281391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4574.637162] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4578.858327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4578.885317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4578.921151] Lustre: 310503:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4579.045683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4580.364411] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4583.356481] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4584.562553] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4586.831317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4586.855629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4586.941198] Lustre: lustre-OST0000: new disk, initializing [ 4586.942497] Lustre: Skipped 1 previous similar message [ 4586.944254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4586.946796] Lustre: Skipped 2 previous similar messages [ 4586.976221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4586.980034] Lustre: Skipped 1 previous similar message [ 4588.275941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4588.278858] Lustre: Skipped 1 previous similar message [ 4588.280797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4588.313722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4588.811888] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4591.757421] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4593.156241] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4593.234315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4594.605201] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4594.696993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4598.754980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4598.755235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4598.755827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4598.757495] LustreError: Skipped 1 previous similar message [ 4598.767587] Lustre: Skipped 1 previous similar message [ 4601.391716] LustreError: 313443:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4601.393821] LustreError: 313443:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4601.430698] Lustre: server umount lustre-OST0000 complete [ 4603.874415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4603.875555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4603.879160] Lustre: Skipped 1 previous similar message [ 4603.881538] Lustre: Skipped 1 previous similar message [ 4608.992313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4608.994161] Lustre: Skipped 2 previous similar messages [ 4614.112542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4614.115534] Lustre: Skipped 1 previous similar message [ 4617.183091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4617.218416] LustreError: 313644:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4617.221017] LustreError: 313644:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4617.247053] Lustre: server umount lustre-MDT0000 complete [ 4618.208167] LustreError: 311300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4618.212450] LustreError: 311300:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4618.399310] LustreError: 311285:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239477 with bad export cookie 13535062284315478147 [ 4618.404242] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4618.522640] Lustre: server umount lustre-MDT0001 complete [ 4621.079709] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:31:20 (1773239480) [ 4621.539639] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4622.017727] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:31:21 (1773239481) [ 4622.932166] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4625.642871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4626.871187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4628.050395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4629.262298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4629.602803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4631.675698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4631.760139] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4631.767222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4631.832498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4632.952932] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4635.814742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4635.852979] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4635.861338] Lustre: 315972:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4637.207665] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4638.918090] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4640.074218] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4642.201768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4642.226679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4642.300893] Lustre: Denying initial registration attempt from nid 192.168.204.142@tcp, specified as failover [ 4642.305337] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4642.308488] LustreError: 317588:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 4642.310864] LustreError: 317588:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4642.312694] LustreError: 317588:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4642.334799] Lustre: server umount lustre-OST0000 complete [ 4642.336179] LustreError: 317588:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4646.367765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4646.369900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4646.373385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4650.558343] LustreError: 317955:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4650.561174] LustreError: 317955:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4650.589829] Lustre: server umount lustre-MDT0000 complete [ 4651.805993] LustreError: 315964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239510 with bad export cookie 13535062284315479015 [ 4651.809574] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4651.811435] LustreError: 315964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4651.915745] Lustre: server umount lustre-MDT0001 complete [ 4655.529410] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 4656.585795] Key type lgssc unregistered [ 4656.699356] LNet: 318690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4656.703246] LNetError: 318690:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4656.715297] LNet: Removed LNI 192.168.204.142@tcp [ 4656.996230] Key type .llcrypt unregistered [ 4656.997280] Key type ._llcrypt unregistered [ 4660.248521] Key type ._llcrypt registered [ 4660.250188] Key type .llcrypt registered [ 4660.288125] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4660.618795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4660.668454] alg: No test for adler32 (adler32-zlib) [ 4661.532380] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4661.623151] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 4663.207180] Key type lgssc registered [ 4663.622034] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4665.832921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4667.020518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4668.176546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4669.387794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4669.731132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4676.063798] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4679.089863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4679.096243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4680.194939] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4680.202256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4680.283522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4681.475887] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4684.240893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4684.271922] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4684.282304] Lustre: 321429:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4684.374654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4685.601717] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4687.272258] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4688.470713] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4690.684566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4690.714785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4690.790417] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4690.864138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4692.744665] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4694.419156] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4697.569181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4697.571328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.572617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4697.574380] Lustre: Skipped 1 previous similar message [ 4701.230108] LustreError: 323715:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4701.265110] Lustre: server umount lustre-OST0000 complete [ 4702.689388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4702.690417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4702.693697] Lustre: Skipped 1 previous similar message [ 4702.696635] Lustre: Skipped 2 previous similar messages [ 4707.809171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4707.811259] Lustre: Skipped 1 previous similar message [ 4708.615739] LustreError: 323916:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4708.618615] LustreError: 323916:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4708.659403] Lustre: server umount lustre-MDT0000 complete [ 4709.935417] LustreError: 321420:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239568 with bad export cookie 14958460930640742199 [ 4709.938344] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4709.942279] LustreError: 321420:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4710.007278] LustreError: 324117:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4710.009176] LustreError: 324117:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4710.072274] Lustre: server umount lustre-MDT0001 complete [ 4713.637535] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 4714.709885] Key type lgssc unregistered [ 4714.833402] LNet: 324648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4714.835690] LNetError: 324648:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4714.842338] LNet: Removed LNI 192.168.204.142@tcp [ 4715.163685] Key type .llcrypt unregistered [ 4715.164719] Key type ._llcrypt unregistered [ 4724.249743] Key type ._llcrypt registered [ 4724.250731] Key type .llcrypt registered [ 4724.293457] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4724.659910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4724.699940] alg: No test for adler32 (adler32-zlib) [ 4725.546162] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4725.626094] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 4727.207233] Key type lgssc registered [ 4727.637918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4730.433325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4730.442137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4731.612500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4732.929383] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4735.996647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4736.162473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4737.423220] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4739.316466] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4740.657406] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4742.986953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4743.101096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4745.221142] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4747.146930] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4761.576568] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4761.580254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4761.584232] Lustre: Skipped 2 previous similar messages [ 4761.587857] Lustre: Skipped 1 previous similar message [ 4762.594959] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.42@tcp (stopping) [ 4762.599461] Lustre: Skipped 1 previous similar message [ 4765.650697] LustreError: 328292:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4765.687164] Lustre: server umount lustre-MDT0000 complete [ 4766.690426] LustreError: 326022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4766.699155] LustreError: 326022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4767.313430] LustreError: 326003:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239626 with bad export cookie 328633926328882751 [ 4767.318736] LustreError: 326003:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4767.318798] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4767.704927] LustreError: 326734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4767.711215] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.42@tcp (stopping) [ 4771.808491] LustreError: 326017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4771.808922] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4771.827952] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4773.459781] LustreError: 328494:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4773.463819] LustreError: 328494:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4773.520006] Lustre: server umount lustre-MDT0001 complete [ 4776.132997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4779.687456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4792.288175] LustreError: 325331:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000edad0888 x1859376470056192/t0(0) o250->MGC192.168.204.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4792.471547] LustreError: 327630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4792.478809] LustreError: 327630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4792.501587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4792.521818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4793.970739] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4797.210093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4797.340273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4798.373711] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4798.654453] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4800.902584] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4802.476410] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4803.517357] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4804.680169] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4805.217318] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4807.653589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4807.660183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4807.660790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4807.667418] Lustre: Skipped 3 previous similar messages [ 4812.591624] LustreError: 330797:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4812.593497] LustreError: 330797:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4812.633924] Lustre: server umount lustre-OST0000 complete [ 4817.891454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4817.891539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4817.892824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4817.907123] Lustre: Skipped 2 previous similar messages [ 4820.057225] LustreError: 330998:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4820.061051] LustreError: 330998:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4820.097431] Lustre: server umount lustre-MDT0000 complete [ 4821.447571] LustreError: 326002:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239680 with bad export cookie 328633926328889912 [ 4821.452852] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4821.596176] Lustre: server umount lustre-MDT0001 complete [ 4825.521424] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 4826.719290] Key type lgssc unregistered [ 4826.877475] LNet: 331730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4826.880168] LNetError: 331730:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4826.889346] LNet: Removed LNI 192.168.204.142@tcp [ 4827.214372] Key type .llcrypt unregistered [ 4827.215798] Key type ._llcrypt unregistered [ 4836.348109] Key type ._llcrypt registered [ 4836.349189] Key type .llcrypt registered [ 4836.391341] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4836.814306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4836.886573] alg: No test for adler32 (adler32-zlib) [ 4837.745759] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4837.829569] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 4839.407174] Key type lgssc registered [ 4839.756253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4842.307507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4842.313802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4843.486287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4844.744847] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4848.156540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4848.348253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4849.655359] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4851.972995] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4853.748684] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4858.146649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4858.153328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4858.156275] Lustre: Skipped 1 previous similar message [ 4861.193450] LustreError: 334495:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4861.222339] Lustre: server umount lustre-MDT0000 complete [ 4862.676136] LustreError: 334497:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239721 with bad export cookie 4619058803269081235 [ 4862.678996] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4862.680200] LustreError: 334497:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4862.756640] LustreError: 334699:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4862.758962] LustreError: 334699:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4862.813298] Lustre: server umount lustre-MDT0001 complete [ 4869.640035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4869.788297] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4869.807619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4869.922277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4871.511280] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4875.088796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4875.125709] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4875.139449] Lustre: 335468:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4875.233717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4876.604656] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4878.381133] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4880.233073] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4884.062456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4884.166822] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4884.172184] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4884.178641] LustreError: 337127:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4884.182268] LustreError: 337127:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4884.186322] LustreError: 337127:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4884.215268] Lustre: server umount lustre-OST0000 complete [ 4884.217041] LustreError: 337127:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4888.364703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4888.488712] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4888.618357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4891.087325] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4893.173049] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4896.663732] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4896.705852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4896.747368] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4896.799955] Lustre: lustre-OST0001: new disk, initializing [ 4896.804066] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4896.868699] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4898.174796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4898.180660] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4898.222148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 4898.254833] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4899.522883] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4901.827949] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4903.397545] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4903.398851] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4903.403058] LustreError: Skipped 1 previous similar message [ 4903.410485] Lustre: Skipped 1 previous similar message [ 4903.411758] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4903.420081] Lustre: Skipped 1 previous similar message [ 4905.140191] LustreError: 339247:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4905.144582] LustreError: 339247:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4905.175867] Lustre: server umount lustre-OST0001 complete [ 4908.516667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4908.517042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4908.518237] LustreError: 337593:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4908.518254] LustreError: 337593:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4908.521577] LustreError: Skipped 1 previous similar message [ 4908.522437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4908.528065] Lustre: Skipped 2 previous similar messages [ 4913.637295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4913.638749] LustreError: 337595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4913.641561] Lustre: Skipped 2 previous similar messages [ 4913.651506] LustreError: 337595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4918.758684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4918.758759] LustreError: 337594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4918.761666] Lustre: Skipped 1 previous similar message [ 4918.772054] LustreError: 337594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4921.311148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4921.337206] LustreError: 339448:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4921.342468] LustreError: 339448:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4921.402165] Lustre: server umount lustre-OST0000 complete [ 4923.874806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4923.882883] Lustre: Skipped 1 previous similar message [ 4928.993290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4928.998410] Lustre: Skipped 3 previous similar messages [ 4937.183118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4937.236733] LustreError: 339651:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4937.240020] LustreError: 339651:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4937.277402] Lustre: server umount lustre-MDT0000 complete [ 4938.676235] LustreError: 335461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239797 with bad export cookie 4619058803269081774 [ 4938.678888] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4938.682134] LustreError: 335461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4938.799648] Lustre: server umount lustre-MDT0001 complete [ 4943.279312] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 4944.486444] Key type lgssc unregistered [ 4944.623537] LNet: 340386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4944.626345] LNetError: 340386:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4944.637519] LNet: Removed LNI 192.168.204.142@tcp [ 4944.974794] Key type .llcrypt unregistered [ 4944.976025] Key type ._llcrypt unregistered [ 4958.410485] Key type ._llcrypt registered [ 4958.412460] Key type .llcrypt registered [ 4958.468118] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 4964.664626] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4965.235678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4965.244482] alg: No test for adler32 (adler32-zlib) [ 4966.107545] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4966.197436] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 4967.783275] Key type lgssc registered [ 4968.209076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4971.096198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4973.149312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4975.408880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4977.500944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4979.764557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4979.788421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4979.793177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4980.880540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4980.890582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4980.925791] Lustre: lustre-MDT0000: new disk, initializing [ 4980.949506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4980.955367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4982.136247] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4986.203854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4986.228311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4986.268342] Lustre: 344309:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4986.291759] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4986.294071] Lustre: Skipped 1 previous similar message [ 4986.332207] Lustre: lustre-MDT0001: new disk, initializing [ 4986.364947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4986.376843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4986.380162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4987.561615] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4990.649541] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4991.819777] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4994.073782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4994.105581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4994.200397] Lustre: lustre-OST0000: new disk, initializing [ 4994.202385] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4994.236832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4995.961236] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4997.110077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4997.113888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4997.139764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4999.100897] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5000.515301] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5000.596313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5001.905799] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5002.009388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5004.721946] LustreError: 347247:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5004.768618] Lustre: server umount lustre-OST0000 complete [ 5006.816343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5006.821118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5006.826697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5007.328241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5007.328810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5012.449105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5012.453462] Lustre: Skipped 2 previous similar messages [ 5015.525955] LustreError: 347448:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5015.529472] LustreError: 347448:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5015.556947] Lustre: server umount lustre-MDT0000 complete [ 5017.005552] LustreError: 347450:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239876 with bad export cookie 11355303741582153438 [ 5017.010981] LustreError: 347450:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5017.011880] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5027.302431] LustreError: 347651:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5027.306982] LustreError: 347651:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5027.388725] Lustre: server umount lustre-MDT0001 complete [ 5030.303623] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 10:38:09 (1773239889) [ 5032.733972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5034.942900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5043.873134] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 5046.651313] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 5049.982610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5052.288749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5054.397369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5056.799465] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5059.571718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5059.593590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5059.675968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5059.687971] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5059.723756] Lustre: lustre-MDT0000: new disk, initializing [ 5059.748567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5059.755364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5061.188010] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5065.415554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5065.440811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5065.469194] Lustre: 351420:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5065.493618] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5065.495220] Lustre: Skipped 1 previous similar message [ 5065.530284] Lustre: lustre-MDT0001: new disk, initializing [ 5065.560576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5065.571852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5065.575482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5066.782423] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5070.199616] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5071.397274] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5074.066323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5074.089129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5074.186057] Lustre: lustre-OST0000: new disk, initializing [ 5074.188617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5074.217480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5075.951487] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5078.960756] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5080.345662] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5082.123263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5082.129272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5082.175082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5082.518300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5084.019320] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5084.101231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5087.204611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5087.206253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5087.210124] LustreError: Skipped 1 previous similar message [ 5087.219908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5087.226878] Lustre: Skipped 1 previous similar message [ 5090.805161] LustreError: 354372:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5090.808974] LustreError: 354372:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5090.874499] Lustre: server umount lustre-OST0000 complete [ 5096.416380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5096.421367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5096.429474] Lustre: Skipped 1 previous similar message [ 5096.433191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5098.773451] LustreError: 354574:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5098.777222] LustreError: 354574:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5098.826628] Lustre: server umount lustre-MDT0000 complete [ 5100.426270] LustreError: 351411:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773239959 with bad export cookie 11355303741582154306 [ 5100.433823] LustreError: 351411:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5100.434105] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5100.580806] Lustre: server umount lustre-MDT0001 complete [ 5103.519794] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 10:39:22 (1773239962) [ 5106.028248] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 10:39:25 (1773239965) [ 5109.914508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5110.145656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5111.733327] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5115.360028] LustreError: 355787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5115.365881] LustreError: 355787:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5115.379432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5116.995638] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5119.199957] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5120.610180] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5123.453973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5125.331497] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5127.058383] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5140.667470] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5141.213116] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5141.739911] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5142.265229] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5143.161305] Lustre: Failing over lustre-MDT0000 [ 5143.328726] LustreError: 358248:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5143.330597] LustreError: 358248:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5143.358607] Lustre: server umount lustre-MDT0000 complete [ 5146.081769] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5146.082955] LustreError: 357943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5146.085726] Lustre: Skipped 3 previous similar messages [ 5146.094014] LustreError: 357943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5147.615160] LustreError: 355781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5151.200779] LustreError: 355786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5151.208100] LustreError: 355786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5156.323160] LustreError: 355786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5156.329661] LustreError: 355786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5156.422280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5156.462251] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5156.562840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5156.564985] Lustre: Skipped 2 previous similar messages [ 5156.591204] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5157.847697] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5157.879669] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5161.958068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5161.965726] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5161.981371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5163.004217] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5163.501605] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5165.308833] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5165.944570] Lustre: Failing over lustre-MDT0000 [ 5166.184184] LustreError: 359328:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5166.186109] LustreError: 359328:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5166.216430] Lustre: server umount lustre-MDT0000 complete [ 5167.073352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5167.074102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5167.074848] LustreError: 357943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5167.079711] Lustre: Skipped 2 previous similar messages [ 5179.020298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5179.055244] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5179.171641] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5180.375324] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5183.449528] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5184.489375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5184.493321] Lustre: Skipped 2 previous similar messages [ 5184.499793] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5184.518242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5185.605783] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5186.109076] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5189.603419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5189.605137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5189.605687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5189.605696] Lustre: Skipped 2 previous similar messages [ 5189.607920] Lustre: Skipped 2 previous similar messages [ 5194.518747] Lustre: server umount lustre-MDT0000 complete [ 5194.721362] LustreError: 357944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5194.726020] LustreError: 357944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 5195.630414] LustreError: 355768:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240054 with bad export cookie 11355303741582156987 [ 5195.633065] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5195.633434] LustreError: 355768:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5195.744456] Lustre: server umount lustre-MDT0001 complete [ 5206.134776] LustreError: 360760:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5206.138165] LustreError: 360760:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5206.194565] Lustre: server umount lustre-OST0000 complete [ 5213.220070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5215.150449] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5218.999807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5220.880651] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5223.323318] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5224.708154] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5227.939472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5228.133271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5228.136749] Lustre: Skipped 3 previous similar messages [ 5230.688533] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5232.536223] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5238.374184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5244.896581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5244.900573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5244.907952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5244.909460] Lustre: Skipped 2 previous similar messages [ 5258.207123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5258.282578] Lustre: server umount lustre-OST0000 complete [ 5265.815221] Lustre: server umount lustre-MDT0000 complete [ 5267.125500] LustreError: 361567:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240126 with bad export cookie 11355303741582157694 [ 5267.129417] LustreError: 361567:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5267.129849] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5267.260439] Lustre: server umount lustre-MDT0001 complete [ 5270.861191] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 5271.910103] Key type lgssc unregistered [ 5272.041305] LNet: 364791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5272.043370] LNetError: 364791:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5272.050325] LNet: Removed LNI 192.168.204.142@tcp [ 5272.340253] Key type .llcrypt unregistered [ 5272.341326] Key type ._llcrypt unregistered [ 5281.677659] Key type ._llcrypt registered [ 5281.678964] Key type .llcrypt registered [ 5281.717322] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 5282.137774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5282.218653] alg: No test for adler32 (adler32-zlib) [ 5283.074043] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 5283.160580] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 5284.743191] Key type lgssc registered [ 5285.124885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5287.647434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5287.653528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5288.801712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5289.965612] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5292.696579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5292.806308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5294.006523] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5295.773730] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5296.912602] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5299.040424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5299.173864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5300.963355] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5302.248063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5302.750691] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5308.384028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5308.386348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5308.390947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5312.480621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5312.482941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5313.726443] LustreError: 368514:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5313.752269] Lustre: server umount lustre-MDT0000 complete [ 5315.003164] LustreError: 366244:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240174 with bad export cookie 4767901038018017978 [ 5315.006717] LustreError: 366244:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5315.006843] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5315.076364] LustreError: 368716:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5315.078791] LustreError: 368716:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5315.124422] Lustre: server umount lustre-MDT0001 complete [ 5326.580775] LustreError: 368916:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5326.584557] LustreError: 368916:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5326.629402] Lustre: server umount lustre-OST0000 complete [ 5331.265994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5331.424132] LustreError: 369646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5331.432859] LustreError: 369646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5331.445829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5332.736225] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5335.645355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5335.771460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5337.057646] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5338.879626] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5340.165694] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5342.433454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5344.477719] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5346.280716] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5357.926832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5371.872061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5371.875346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5371.880906] Lustre: Skipped 1 previous similar message [ 5371.883579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5371.886295] Lustre: Skipped 2 previous similar messages [ 5375.038170] LustreError: 372014:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5375.040535] LustreError: 372014:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5375.066225] Lustre: server umount lustre-MDT0000 complete [ 5376.361206] LustreError: 369627:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240235 with bad export cookie 4767901038018019077 [ 5376.364910] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5376.365721] LustreError: 369627:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5376.487171] Lustre: server umount lustre-MDT0001 complete [ 5388.017687] LustreError: 372418:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5388.019402] LustreError: 372418:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5388.057752] Lustre: server umount lustre-OST0000 complete [ 5393.071272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5393.232602] LustreError: 373247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5393.253646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5393.257136] Lustre: Skipped 1 previous similar message [ 5394.586378] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5397.396598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5398.795468] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5400.559931] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5401.688117] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5403.913422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5405.862655] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5407.534878] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5418.341834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5427.679894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5427.682812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5427.686750] Lustre: Skipped 2 previous similar messages [ 5427.689378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5427.691086] Lustre: Skipped 3 previous similar messages [ 5430.252884] LustreError: 375516:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5430.254890] LustreError: 375516:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5430.291139] Lustre: server umount lustre-OST0000 complete [ 5433.311890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5433.314513] LustreError: Skipped 1 previous similar message [ 5433.315894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5433.319845] Lustre: Skipped 1 previous similar message [ 5433.321785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5433.323719] Lustre: Skipped 2 previous similar messages [ 5437.650848] Lustre: server umount lustre-MDT0000 complete [ 5438.885184] LustreError: 373228:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240297 with bad export cookie 4767901038018020085 [ 5438.888783] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5438.888879] LustreError: 373228:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5439.013076] Lustre: server umount lustre-MDT0001 complete [ 5442.697767] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 5443.769167] Key type lgssc unregistered [ 5443.915356] LNet: 376449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5443.917960] LNetError: 376449:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5443.924347] LNet: Removed LNI 192.168.204.142@tcp [ 5444.252041] Key type .llcrypt unregistered [ 5444.253971] Key type ._llcrypt unregistered [ 5453.687242] Key type ._llcrypt registered [ 5453.688559] Key type .llcrypt registered [ 5453.722765] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 5454.019657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5454.056618] alg: No test for adler32 (adler32-zlib) [ 5454.901693] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 5454.979338] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 5456.559112] Key type lgssc registered [ 5456.942267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5459.495061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5459.501603] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5459.505698] LustreError: 377846:0:(osd_handler.c:8572:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5459.515181] LustreError: 377846:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5459.518798] LustreError: 377846:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5459.522450] LustreError: 377846:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5459.526696] LustreError: 377846:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5462.032231] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5462.037059] LustreError: 378243:0:(osd_handler.c:8572:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5462.048102] LustreError: 378243:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5462.050434] LustreError: 378243:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5462.052721] LustreError: 378243:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5462.055185] LustreError: 378243:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5468.508062] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 5469.481651] Key type lgssc unregistered [ 5469.604372] LNet: 379228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5469.606787] LNetError: 379228:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5469.619329] LNet: Removed LNI 192.168.204.142@tcp [ 5469.929407] Key type .llcrypt unregistered [ 5469.930540] Key type ._llcrypt unregistered [ 5476.099923] Key type ._llcrypt registered [ 5476.101507] Key type .llcrypt registered [ 5476.148217] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 5482.284727] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 5482.695970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5482.751783] alg: No test for adler32 (adler32-zlib) [ 5483.617719] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 5483.706490] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 5485.287162] Key type lgssc registered [ 5485.682948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5488.697602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5490.870129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5492.821548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5494.887547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5497.196796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5497.210460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5497.216073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5498.317182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5498.331533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5498.373306] Lustre: lustre-MDT0000: new disk, initializing [ 5498.400061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5498.406719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5499.756913] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5503.979056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5504.003180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5504.037056] Lustre: 382330:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5504.059371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5504.061022] Lustre: Skipped 1 previous similar message [ 5504.103806] Lustre: lustre-MDT0001: new disk, initializing [ 5504.141768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5504.158167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5504.163028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5505.438853] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5508.517780] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5509.631212] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5512.003775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5512.035898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5512.154418] Lustre: lustre-OST0000: new disk, initializing [ 5512.156312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5512.187484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5514.125281] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5517.316711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5517.320829] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5517.336439] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5517.371040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5518.726098] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5518.803942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5520.084988] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5520.151811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5522.401620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5522.403201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5522.408883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5522.412099] Lustre: Skipped 1 previous similar message [ 5526.829147] LustreError: 385267:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5526.869799] Lustre: server umount lustre-OST0000 complete [ 5530.079957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5530.084035] LustreError: Skipped 1 previous similar message [ 5530.086401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5530.092029] Lustre: Skipped 1 previous similar message [ 5530.095159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5532.640210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5532.640719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5534.272970] LustreError: 385468:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5534.275327] LustreError: 385468:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5534.306868] Lustre: server umount lustre-MDT0000 complete [ 5535.601175] LustreError: 383113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240394 with bad export cookie 14169259706972846411 [ 5535.604473] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5535.604829] LustreError: 383113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5535.680471] LustreError: 385670:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5535.684127] LustreError: 385670:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5535.739777] Lustre: server umount lustre-MDT0001 complete [ 5538.498605] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5538.995897] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 10:46:38 (1773240398) [ 5541.128243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.304693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5542.587526] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5545.467741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5545.598970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5546.957563] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5548.815305] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5550.043918] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5552.219451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5554.103100] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5555.811223] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5557.972852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5557.996075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5558.046612] Lustre: lustre-OST0001: new disk, initializing [ 5558.048430] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5558.073826] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5558.075916] Lustre: Skipped 1 previous similar message [ 5559.823170] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5562.827356] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5564.423889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5564.427116] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5564.466111] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5569.504365] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5569.504915] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5569.507953] LustreError: Skipped 1 previous similar message [ 5569.515114] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5569.518227] Lustre: Skipped 2 previous similar messages [ 5571.311377] LustreError: 389589:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5571.313343] LustreError: 389589:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5571.334820] Lustre: server umount lustre-OST0001 complete [ 5572.080533] LustreError: 388494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5572.087801] LustreError: 388494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5572.583538] LustreError: 388494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5572.587805] LustreError: 388494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5577.698977] LustreError: 387989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5577.704152] LustreError: 387989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5582.825094] LustreError: 388494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5582.831353] LustreError: 388494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5586.400321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5586.402655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5586.406145] Lustre: Skipped 1 previous similar message [ 5586.408161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5587.940120] LustreError: 387987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5587.947711] LustreError: 387987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5589.550805] LustreError: 389812:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5589.553436] LustreError: 389812:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5589.588030] Lustre: server umount lustre-OST0000 complete [ 5592.031863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5592.035639] LustreError: Skipped 1 previous similar message [ 5598.176124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5598.177717] Lustre: Skipped 6 previous similar messages [ 5605.343102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5605.374647] LustreError: 390013:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5605.376394] LustreError: 390013:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5605.404361] Lustre: server umount lustre-MDT0000 complete [ 5606.581101] LustreError: 386368:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240465 with bad export cookie 14169259706972847265 [ 5606.584021] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5606.584488] LustreError: 386368:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5612.782660] Lustre: server umount lustre-MDT0001 complete [ 5616.348752] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 5617.300592] Key type lgssc unregistered [ 5617.418511] LNet: 390749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5617.422872] LNetError: 390749:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5617.432430] LNet: Removed LNI 192.168.204.142@tcp [ 5617.728734] Key type .llcrypt unregistered [ 5617.729834] Key type ._llcrypt unregistered [ 5628.338039] Key type ._llcrypt registered [ 5628.339443] Key type .llcrypt registered [ 5628.371448] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 5634.767513] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 5635.139371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5635.189710] alg: No test for adler32 (adler32-zlib) [ 5636.044271] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 5636.126783] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 5637.703124] Key type lgssc registered [ 5638.031916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5640.969765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5643.226420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5645.387865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5647.333251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5649.462667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5649.481833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5649.487267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5650.579278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5650.590123] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5650.625414] Lustre: lustre-MDT0000: new disk, initializing [ 5650.649441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5650.655554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5651.878397] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5655.871133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5655.894044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5655.927793] Lustre: 394670:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5655.952665] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5655.954341] Lustre: Skipped 1 previous similar message [ 5655.989629] Lustre: lustre-MDT0001: new disk, initializing [ 5656.027811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5656.039870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5656.042893] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5657.244546] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5660.121567] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5661.245414] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5663.302539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5663.334257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5663.434862] Lustre: lustre-OST0000: new disk, initializing [ 5663.436726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5663.469763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5664.063135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5664.066260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5664.115290] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5665.223147] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5668.126629] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5669.365156] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5669.432824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5670.598521] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5670.685755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5674.468878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5674.469593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5674.470939] LustreError: Skipped 1 previous similar message [ 5674.474496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5674.477179] Lustre: Skipped 1 previous similar message [ 5677.357122] LustreError: 397606:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5677.391730] Lustre: server umount lustre-OST0000 complete [ 5679.585653] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5679.586675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5679.588728] Lustre: Skipped 2 previous similar messages [ 5684.668496] LustreError: 397808:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5684.670271] LustreError: 397808:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5684.696382] Lustre: server umount lustre-MDT0000 complete [ 5685.812487] LustreError: 394662:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240544 with bad export cookie 4355953960992839387 [ 5685.815663] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5685.818170] LustreError: 394662:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5685.877342] LustreError: 398009:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5685.879066] LustreError: 398009:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5685.937844] Lustre: server umount lustre-MDT0001 complete [ 5688.554145] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 10:49:07 (1773240547) [ 5691.360189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5693.936240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5694.073475] Lustre: lustre-MDT0000: new disk, initializing [ 5694.083842] LustreError: 399276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5694.088406] LustreError: 399276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5694.109586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5695.307025] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5697.947921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5699.244378] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5700.802814] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5701.832788] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5702.433396] LustreError: 399307:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5702.436472] LustreError: 399307:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5702.437203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5702.437885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5702.437903] Lustre: Skipped 1 previous similar message [ 5702.438932] LustreError: 399307:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 5702.443129] Lustre: Skipped 1 previous similar message [ 5707.744236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5707.746338] Lustre: Skipped 2 previous similar messages [ 5708.618216] LustreError: 400618:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5708.620198] LustreError: 400618:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5708.646669] Lustre: server umount lustre-MDT0000 complete [ 5708.768401] LustreError: 399276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5708.772409] LustreError: 399276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5709.793867] LustreError: 399255:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240568 with bad export cookie 4355953960992840255 [ 5709.799692] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5709.918229] Lustre: server umount lustre-MDT0001 complete [ 5712.619938] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 10:49:31 (1773240571) [ 5714.622710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5714.776353] LustreError: 401538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.797728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5714.799572] Lustre: Skipped 1 previous similar message [ 5716.003452] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5718.630532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5719.974489] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5721.643471] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5722.734409] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5724.898359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5725.055705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5725.059531] Lustre: Skipped 1 previous similar message [ 5727.330558] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5729.410640] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5732.939863] Lustre: Permanently deactivating lustre-OST0000 [ 5732.944424] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5734.703556] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5736.028315] LustreError: 404081:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5736.031817] LustreError: 404081:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 5736.033935] LustreError: 404081:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5736.580118] LustreError: 404130:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 5736.582403] LustreError: 404130:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5743.213655] LustreError: 404228:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5743.215561] LustreError: 404228:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5743.246869] Lustre: server umount lustre-OST0000 complete [ 5744.030113] LustreError: 404379:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 5744.032316] LustreError: 404379:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5744.608687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5744.610977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5744.614596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5748.704666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5753.824755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5753.826437] Lustre: Skipped 3 previous similar messages [ 5754.850645] LustreError: 404477:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5754.852474] LustreError: 404477:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5754.873718] Lustre: server umount lustre-MDT0000 complete [ 5755.967413] LustreError: 403155:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240614 with bad export cookie 4355953960992840773 [ 5755.970594] LustreError: 403155:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5755.972111] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5765.132439] Lustre: server umount lustre-MDT0001 complete [ 5767.588592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5768.694923] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5769.464495] LustreError: 405520:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5771.267114] Lustre: 405858:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.142@tcp [ 5771.834572] LustreError: 405955:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5771.836537] LustreError: 405955:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5771.858879] Lustre: server umount MGS complete [ 5775.174856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5775.325207] LustreError: 406584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5775.330592] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5775.332613] Lustre: Skipped 1 previous similar message [ 5775.350273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5776.444525] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5778.890989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5778.980801] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5780.086448] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5781.580113] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5782.666709] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5783.239441] Lustre: Permanently reactivating lustre-OST0000 [ 5783.243757] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5783.245551] Lustre: Skipped 2 previous similar messages [ 5811.679201] LustreError: 408284:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5815.381193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5815.506349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5815.508353] Lustre: Skipped 1 previous similar message [ 5817.491382] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5819.274909] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5832.162861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5832.163161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5832.163643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5832.166120] Lustre: Skipped 1 previous similar message [ 5832.169635] LustreError: Skipped 1 previous similar message [ 5836.972948] LustreError: 410110:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5836.974796] LustreError: 410110:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5837.010452] Lustre: server umount lustre-OST0000 complete [ 5840.352139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5844.303619] Lustre: server umount lustre-MDT0000 complete [ 5845.416228] LustreError: 406564:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240704 with bad export cookie 4355953960992841704 [ 5845.419080] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5845.419419] LustreError: 406564:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5845.531182] Lustre: server umount lustre-MDT0001 complete [ 5848.697368] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 5849.647068] Key type lgssc unregistered [ 5849.763361] LNet: 411048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5849.765484] LNetError: 411048:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5849.772322] LNet: Removed LNI 192.168.204.142@tcp [ 5850.046059] Key type .llcrypt unregistered [ 5850.047084] Key type ._llcrypt unregistered [ 5855.630865] Key type ._llcrypt registered [ 5855.631880] Key type .llcrypt registered [ 5855.664820] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 5861.949455] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 5862.343394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5862.396321] alg: No test for adler32 (adler32-zlib) [ 5863.259793] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 5863.347197] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 5864.927122] Key type lgssc registered [ 5865.271463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5867.937444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5869.879418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5871.751412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5873.848971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5876.021652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5876.047642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5876.053859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5877.154783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5877.166165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5877.204865] Lustre: lustre-MDT0000: new disk, initializing [ 5877.230848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5877.237598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5878.556847] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5882.648612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5882.671373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5882.700937] Lustre: 414154:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5882.723386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5882.724919] Lustre: Skipped 1 previous similar message [ 5882.762650] Lustre: lustre-MDT0001: new disk, initializing [ 5882.792653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5882.804032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5882.807032] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5883.976728] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5886.913775] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5888.042489] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5890.251671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5890.277389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5890.377896] Lustre: lustre-OST0000: new disk, initializing [ 5890.379597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5890.411451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5890.742822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5890.746568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5890.779826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5892.276718] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5895.159251] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5896.315344] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5896.380756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5897.488604] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5897.551429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5901.283180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5901.283807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5901.284553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5901.288020] Lustre: Skipped 1 previous similar message [ 5904.173548] LustreError: 417094:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5904.206425] Lustre: server umount lustre-OST0000 complete [ 5906.401161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5906.401834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5906.405553] Lustre: Skipped 1 previous similar message [ 5906.409435] Lustre: Skipped 1 previous similar message [ 5911.488454] LustreError: 417295:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5911.490381] LustreError: 417295:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5911.517750] Lustre: server umount lustre-MDT0000 complete [ 5912.695098] LustreError: 414147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240771 with bad export cookie 14523617112577076053 [ 5912.698350] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5912.701026] LustreError: 414147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5912.761419] LustreError: 417497:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5912.763248] LustreError: 417497:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5912.813238] Lustre: server umount lustre-MDT0001 complete [ 5915.303313] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5915.836496] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 10:52:55 (1773240775) [ 5917.989187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5918.131930] LustreError: 418215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5918.136933] LustreError: 418215:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5918.152426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5919.293603] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5921.818522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5923.036238] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5924.574811] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5925.594118] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5927.565807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5927.684836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5927.688471] Lustre: Skipped 1 previous similar message [ 5929.387938] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5930.958369] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5931.734582] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5933.186360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5937.121682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5937.122332] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5937.123766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5937.123771] Lustre: Skipped 1 previous similar message [ 5937.126597] Lustre: Skipped 1 previous similar message [ 5937.128531] LustreError: Skipped 1 previous similar message [ 5940.527189] LustreError: 420651:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5940.529012] LustreError: 420651:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5940.561612] Lustre: server umount lustre-OST0000 complete [ 5942.240478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5942.241306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5942.243746] Lustre: Skipped 1 previous similar message [ 5942.245364] Lustre: Skipped 1 previous similar message [ 5947.359909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5947.361593] Lustre: Skipped 1 previous similar message [ 5947.834784] LustreError: 420852:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5947.836501] LustreError: 420852:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5947.862451] Lustre: server umount lustre-MDT0000 complete [ 5948.972188] LustreError: 419834:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240807 with bad export cookie 14523617112577076921 [ 5948.974886] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5948.975187] LustreError: 419834:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5949.081572] Lustre: server umount lustre-MDT0001 complete [ 5952.088682] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 5953.075918] Key type lgssc unregistered [ 5953.202304] LNet: 421585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5953.204350] LNetError: 421585:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5953.214385] LNet: Removed LNI 192.168.204.142@tcp [ 5953.496269] Key type .llcrypt unregistered [ 5953.497248] Key type ._llcrypt unregistered [ 5960.108168] Key type ._llcrypt registered [ 5960.109183] Key type .llcrypt registered [ 5960.139714] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 5960.221164] Key type .llcrypt unregistered [ 5960.222149] Key type ._llcrypt unregistered [ 5966.459954] Key type ._llcrypt registered [ 5966.461024] Key type .llcrypt registered [ 5966.492152] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 5966.743405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5966.760444] alg: No test for adler32 (adler32-zlib) [ 5967.598770] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 5967.671478] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 5969.247119] Key type lgssc registered [ 5969.550973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5971.715365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5971.720586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5972.863597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5973.930430] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5976.371548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5976.492080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5978.209248] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5979.784374] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5981.664931] LustreError: 423733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5981.668902] LustreError: 423733:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5981.845541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5981.981126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5983.016474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5983.151374] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5984.917084] Lustre: 423728:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 5984.919847] Lustre: 423728:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5984.921928] Lustre: 423728:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 5984.923736] Lustre: 423728:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/153/0, punch: 0/0/0, quota 1/3/0 [ 5984.925707] Lustre: 423728:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5984.927484] Lustre: 423728:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5987.296192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5987.298231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5987.301578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5988.321718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5988.322369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5988.326281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5991.852221] LustreError: 425786:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5991.884675] Lustre: server umount lustre-OST0000 complete [ 5993.440249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5993.440830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5993.443609] Lustre: Skipped 1 previous similar message [ 5993.445281] Lustre: Skipped 1 previous similar message [ 5998.560139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5998.562192] Lustre: Skipped 1 previous similar message [ 5999.226974] LustreError: 425988:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5999.228692] LustreError: 425988:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5999.253281] Lustre: server umount lustre-MDT0000 complete [ 6000.293100] LustreError: 423713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240859 with bad export cookie 17348659601809724768 [ 6000.296079] LustreError: 423713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6000.296586] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6000.362857] LustreError: 426192:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6000.364534] LustreError: 426192:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6000.407361] Lustre: server umount lustre-MDT0001 complete [ 6003.339444] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6004.160878] Key type lgssc unregistered [ 6004.273301] LNet: 426723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6004.275365] LNetError: 426723:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6004.281346] LNet: Removed LNI 192.168.204.142@tcp [ 6004.526903] Key type .llcrypt unregistered [ 6004.527994] Key type ._llcrypt unregistered [ 6012.701173] Key type ._llcrypt registered [ 6012.702787] Key type .llcrypt registered [ 6012.736058] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6013.030045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6013.078219] alg: No test for adler32 (adler32-zlib) [ 6013.945928] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 6014.027653] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6015.599097] Key type lgssc registered [ 6015.893615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6018.835360] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6018.844351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6034.335303] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6039.455138] LustreError: 428084:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.142@tcp: failed processing log, type 4: rc = -110 [ 6066.079110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6067.712713] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6069.524111] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6071.580944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6071.736323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6071.753298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6072.802670] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6075.189147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6075.301701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6076.356901] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6077.930617] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6079.018587] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6084.677251] Lustre: 429005:0:(osd_internal.h:1471:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 6084.680041] Lustre: 429005:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6084.681981] Lustre: 429005:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 6084.683884] Lustre: 429005:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 6084.685812] Lustre: 429005:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6084.687673] Lustre: 429005:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6085.601071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6085.603201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6085.606102] Lustre: Skipped 1 previous similar message [ 6085.609759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6090.722384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6090.724367] Lustre: Skipped 2 previous similar messages [ 6091.627177] LustreError: 430364:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 6091.662190] Lustre: server umount lustre-OST0000 complete [ 6095.840713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6095.840770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6095.841269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6095.843700] Lustre: Skipped 1 previous similar message [ 6098.747024] LustreError: 430566:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6098.748800] LustreError: 430566:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6098.774764] Lustre: server umount lustre-MDT0000 complete [ 6099.887073] LustreError: 430568:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240958 with bad export cookie 14823565885305051077 [ 6099.889755] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6099.890231] LustreError: 430568:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6099.950840] LustreError: 430770:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6099.952561] LustreError: 430770:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6099.996429] Lustre: server umount lustre-MDT0001 complete [ 6103.191649] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6104.016689] Key type lgssc unregistered [ 6104.132271] LNet: 431301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6104.134368] LNetError: 431301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6104.141314] LNet: Removed LNI 192.168.204.142@tcp [ 6104.388034] Key type .llcrypt unregistered [ 6104.389105] Key type ._llcrypt unregistered [ 6112.545303] Key type ._llcrypt registered [ 6112.546341] Key type .llcrypt registered [ 6112.576327] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6112.944333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6112.954341] alg: No test for adler32 (adler32-zlib) [ 6113.790981] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 6113.860522] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6115.431099] Key type lgssc registered [ 6115.713513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6117.859712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6117.864971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6119.006062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6120.055432] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6122.439747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6122.549076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6123.604209] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6125.111208] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6126.110172] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6128.021596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6128.127321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6129.192060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6129.698613] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6131.189835] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6137.313034] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6137.314476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6137.316155] Lustre: Skipped 2 previous similar messages [ 6137.317585] Lustre: Skipped 1 previous similar message [ 6137.820810] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.42@tcp (stopping) [ 6137.822586] Lustre: Skipped 1 previous similar message [ 6138.999035] LustreError: 434918:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6139.022183] Lustre: server umount lustre-MDT0000 complete [ 6139.611409] LustreError: 432657:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.42@tcp arrived at 1773240998 with bad export cookie 12566599375264352764 [ 6139.614698] LustreError: 432657:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6140.118120] LustreError: 432657:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773240999 with bad export cookie 12566599375264351917 [ 6140.120917] LustreError: 432657:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6140.121093] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6142.432081] LustreError: 432675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6142.432669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6142.432741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6142.433372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6142.436239] LustreError: 432675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6145.439812] LustreError: 432675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6145.440096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6145.443975] LustreError: 432675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6146.221458] LustreError: 435121:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6146.223334] LustreError: 435121:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6146.254343] Lustre: server umount lustre-OST0000 complete [ 6148.111863] LustreError: 435475:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6148.113593] LustreError: 435475:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6148.157425] Lustre: server umount lustre-MDT0001 complete [ 6151.112536] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6151.926905] Key type lgssc unregistered [ 6152.050343] LNet: 436005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6152.052415] LNetError: 436005:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6152.062257] LNet: Removed LNI 192.168.204.142@tcp [ 6152.304503] Key type .llcrypt unregistered [ 6152.305531] Key type ._llcrypt unregistered [ 6160.956910] Key type ._llcrypt registered [ 6160.958616] Key type .llcrypt registered [ 6160.990099] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6161.354994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6161.361661] alg: No test for adler32 (adler32-zlib) [ 6162.203294] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 6162.274449] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6163.847166] Key type lgssc registered [ 6164.165380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6166.371950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6166.377292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6167.520159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6168.630592] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6171.103980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6171.213979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6172.311296] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6173.895157] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6174.966383] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6176.968179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6177.071902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6178.086406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6178.659598] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6180.234567] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6186.461238] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.42@tcp (stopping) [ 6186.465560] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6186.465727] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6186.469877] Lustre: Skipped 1 previous similar message [ 6188.152110] LustreError: 439621:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6188.169373] Lustre: server umount lustre-MDT0001 complete [ 6191.584156] LustreError: 437379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6191.584820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6191.587910] LustreError: 437379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6191.591150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6191.593986] Lustre: Skipped 2 previous similar messages [ 6195.563979] LustreError: 439826:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6195.565610] LustreError: 439826:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6195.592099] Lustre: server umount lustre-OST0000 complete [ 6196.641075] LustreError: 440027:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6196.642668] LustreError: 440027:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6196.684215] Lustre: server umount lustre-MDT0000 complete [ 6200.165747] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6200.918997] Key type lgssc unregistered [ 6201.020287] LNet: 440709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6201.022229] LNetError: 440709:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6201.029243] LNet: Removed LNI 192.168.204.142@tcp [ 6201.243025] Key type .llcrypt unregistered [ 6201.243969] Key type ._llcrypt unregistered [ 6207.984543] Key type ._llcrypt registered [ 6207.985562] Key type .llcrypt registered [ 6208.017878] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 6212.986696] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6213.230975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6213.263840] alg: No test for adler32 (adler32-zlib) [ 6214.094173] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 6214.155883] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6215.719095] Key type lgssc registered [ 6215.968071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6218.101191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6219.697827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6221.216535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6222.787200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6225.127454] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6227.658090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6229.584933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6229.601813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6229.605798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6230.680249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6230.689617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6230.723663] Lustre: lustre-MDT0000: new disk, initializing [ 6230.744875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6230.750326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6231.830481] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6235.007570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6236.913364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6236.935019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6237.018193] Lustre: lustre-OST0000: new disk, initializing [ 6237.019906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6237.021569] Lustre: Skipped 1 previous similar message [ 6237.040786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6238.611929] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6239.929746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6239.932528] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6239.939601] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6241.374746] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6244.254938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6246.223198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6246.244624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6246.275125] Lustre: 444655:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6246.299324] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6246.333440] Lustre: lustre-MDT0001: new disk, initializing [ 6246.371211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6246.383873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6246.386638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6247.492368] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6250.682057] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6250.743699] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6251.834567] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6251.896261] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6255.288449] LustreError: 448078:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6255.306141] Lustre: server umount lustre-MDT0001 complete [ 6257.633332] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6257.633489] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6262.753499] LustreError: 445486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6262.758819] LustreError: 445486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6267.872231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6267.872361] LustreError: 444666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6267.876085] Lustre: Skipped 1 previous similar message [ 6267.882585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6272.992332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6272.992817] LustreError: 444666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6273.258198] LustreError: 448348:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6273.260013] LustreError: 448348:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6273.289496] Lustre: server umount lustre-OST0000 complete [ 6280.570890] LustreError: 448549:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6280.572697] LustreError: 448549:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6280.612081] Lustre: server umount lustre-MDT0000 complete [ 6284.217846] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6284.998550] Key type lgssc unregistered [ 6285.113309] LNet: 449233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6285.115439] LNetError: 449233:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6285.123343] LNet: Removed LNI 192.168.204.142@tcp [ 6285.374879] Key type .llcrypt unregistered [ 6285.375905] Key type ._llcrypt unregistered [ 6297.089084] Key type ._llcrypt registered [ 6297.090118] Key type .llcrypt registered [ 6297.125648] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6297.214229] Key type .llcrypt unregistered [ 6297.215173] Key type ._llcrypt unregistered [ 6303.253722] Key type ._llcrypt registered [ 6303.254699] Key type .llcrypt registered [ 6303.284658] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6303.529760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6303.568591] alg: No test for adler32 (adler32-zlib) [ 6304.406941] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 6304.476682] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6306.047153] Key type lgssc registered [ 6306.344164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6308.534179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6310.265794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6311.961324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6313.880172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6313.903494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6313.907313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6314.978933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6314.987732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6315.028213] Lustre: lustre-MDT0000: new disk, initializing [ 6315.049524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6315.054725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6316.117573] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6319.718745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6319.746499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6319.771621] Lustre: 452549:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6319.792848] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6319.794189] Lustre: Skipped 1 previous similar message [ 6319.822163] Lustre: lustre-MDT0001: new disk, initializing [ 6319.853040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6319.863873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6319.867309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6320.924860] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6323.616747] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6324.865566] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6327.058232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6327.088185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6327.180482] Lustre: lustre-OST0000: new disk, initializing [ 6327.182287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6327.211929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6328.855378] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6331.626921] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6331.898395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6331.901194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6331.939403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6340.575945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6340.578116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6340.581589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6342.114951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6342.115377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6342.119275] Lustre: Skipped 2 previous similar messages [ 6345.021513] LustreError: 454991:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6345.049787] Lustre: server umount lustre-MDT0000 complete [ 6346.100097] LustreError: 452540:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773241205 with bad export cookie 4797926114828227730 [ 6346.103344] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6346.103530] LustreError: 452540:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6346.179673] LustreError: 455193:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6346.183433] LustreError: 455193:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6346.245338] Lustre: server umount lustre-MDT0001 complete [ 6357.616852] LustreError: 455395:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6357.618697] LustreError: 455395:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6357.660214] Lustre: server umount lustre-OST0000 complete [ 6360.523637] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:00:19 (1773241219) [ 6360.824978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6362.835126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6362.997309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6364.134148] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6366.598017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6367.801698] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6369.356364] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6370.360059] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6372.391425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6372.420208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6372.509140] Lustre: Found index 0 for lustre-OST0000, updating log [ 6372.581157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6372.582825] Lustre: Skipped 1 previous similar message [ 6373.610763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6374.270807] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6375.823701] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6378.975775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6378.977891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6378.981122] Lustre: Skipped 1 previous similar message [ 6378.982782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6382.893099] LustreError: 458469:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6382.894735] LustreError: 458469:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6382.927930] Lustre: server umount lustre-OST0000 complete [ 6383.765418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6385.377458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6385.380526] Lustre: Skipped 1 previous similar message [ 6385.382848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6385.384351] Lustre: Skipped 4 previous similar messages [ 6390.460688] LustreError: 458732:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6390.462367] LustreError: 458732:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6390.490390] Lustre: server umount lustre-MDT0000 complete [ 6390.752438] LustreError: 456220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6390.756768] LustreError: 456220:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6391.659684] LustreError: 456202:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773241250 with bad export cookie 4797926114828229109 [ 6391.663834] LustreError: 456202:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6391.664559] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6391.821820] Lustre: server umount lustre-MDT0001 complete [ 6394.494173] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 11:00:53 (1773241253) [ 6394.820901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6395.133594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6395.445572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6395.751205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6396.055142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6396.364424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6396.674201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6397.002746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6397.451861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6397.965087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6398.477598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6399.004541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6399.529483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6400.042551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6400.524657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6400.945246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6401.259330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6401.594442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6401.912508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6402.213091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6402.532794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6402.853453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6403.163223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6403.485461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6403.790805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6404.107585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6404.420426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6404.733444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6405.136453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6405.622065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6406.087862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6406.559101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6407.020135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6407.496125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6407.979865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6408.457156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6408.967787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6409.479069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6409.981549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6410.487984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6410.982707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6411.485812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6411.981402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6412.479680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6412.993361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6413.499169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6413.996788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6414.492662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6415.004696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6415.518889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6416.031914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6419.791598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6420.044508] LustreError: 462329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6420.093127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6422.024620] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6425.569573] LustreError: 462329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6426.278611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6428.212565] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6430.382726] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6432.067427] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6435.364982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6435.410284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6435.542223] Lustre: Found index 0 for lustre-OST0000, updating log [ 6438.233816] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6440.815875] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6442.980328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6442.985616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6442.994427] Lustre: Skipped 1 previous similar message [ 6442.998240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6447.919361] LustreError: 464639:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6447.923613] LustreError: 464639:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6447.984763] Lustre: server umount lustre-OST0000 complete [ 6451.874626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6451.878422] Lustre: Skipped 5 previous similar messages [ 6456.062319] Lustre: server umount lustre-MDT0000 complete [ 6456.994203] LustreError: 463490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6457.005207] LustreError: 463490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6457.944403] LustreError: 462308:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773241316 with bad export cookie 4797926114828230173 [ 6457.951310] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6457.957410] LustreError: 462308:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6458.097708] Lustre: server umount lustre-MDT0001 complete [ 6459.529715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6467.350599] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 6471.967429] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6476.973308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6480.393345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6483.577634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6486.887230] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6489.880592] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 11:02:28 (1773241348) [ 6511.333197] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6512.990595] Key type lgssc unregistered [ 6513.164617] LNet: 468634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6513.169506] LNetError: 468634:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6637.096408] LNet: Removed LNI 192.168.204.142@tcp [ 6637.526388] Key type .llcrypt unregistered [ 6637.528642] Key type ._llcrypt unregistered [ 6645.768643] Key type ._llcrypt registered [ 6645.770907] Key type .llcrypt registered [ 6645.850535] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6646.199839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6646.206206] alg: No test for adler32 (adler32-zlib) [ 6647.103334] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 6647.212935] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6648.807205] Key type lgssc registered [ 6649.379469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6651.282715] LNet: Added LNI 192.168.204.142@tcp100 [8/256/0/180] [ 6651.544678] LNet: Added LNI 192.168.204.142@tcp101 [8/256/0/180] [ 6651.806493] LNet: Added LNI 192.168.204.142@tcp102 [8/256/0/180] [ 6652.068913] LNet: Added LNI 192.168.204.142@tcp103 [8/256/0/180] [ 6652.332150] LNet: Added LNI 192.168.204.142@tcp104 [8/256/0/180] [ 6652.596575] LNet: Added LNI 192.168.204.142@tcp105 [8/256/0/180] [ 6652.858958] LNet: Added LNI 192.168.204.142@tcp106 [8/256/0/180] [ 6653.123799] LNet: Added LNI 192.168.204.142@tcp107 [8/256/0/180] [ 6653.384964] LNet: Added LNI 192.168.204.142@tcp108 [8/256/0/180] [ 6653.647128] LNet: Added LNI 192.168.204.142@tcp109 [8/256/0/180] [ 6653.907151] LNet: Added LNI 192.168.204.142@tcp110 [8/256/0/180] [ 6654.165918] LNet: Added LNI 192.168.204.142@tcp111 [8/256/0/180] [ 6654.423840] LNet: Added LNI 192.168.204.142@tcp112 [8/256/0/180] [ 6654.681852] LNet: Added LNI 192.168.204.142@tcp113 [8/256/0/180] [ 6654.945785] LNet: Added LNI 192.168.204.142@tcp114 [8/256/0/180] [ 6655.218766] LNet: Added LNI 192.168.204.142@tcp115 [8/256/0/180] [ 6655.560904] LNet: Added LNI 192.168.204.142@tcp116 [8/256/0/180] [ 6655.874745] LNet: Added LNI 192.168.204.142@tcp117 [8/256/0/180] [ 6656.222836] LNet: Added LNI 192.168.204.142@tcp118 [8/256/0/180] [ 6656.530039] LNet: Added LNI 192.168.204.142@tcp119 [8/256/0/180] [ 6656.790247] LNet: Added LNI 192.168.204.142@tcp120 [8/256/0/180] [ 6657.051348] LNet: Added LNI 192.168.204.142@tcp121 [8/256/0/180] [ 6657.315551] LNet: Added LNI 192.168.204.142@tcp122 [8/256/0/180] [ 6657.580154] LNet: Added LNI 192.168.204.142@tcp123 [8/256/0/180] [ 6657.839183] LNet: Added LNI 192.168.204.142@tcp124 [8/256/0/180] [ 6658.105217] LNet: Added LNI 192.168.204.142@tcp125 [8/256/0/180] [ 6658.360379] LNet: Added LNI 192.168.204.142@tcp126 [8/256/0/180] [ 6658.617084] LNet: Added LNI 192.168.204.142@tcp127 [8/256/0/180] [ 6658.874745] LNet: Added LNI 192.168.204.142@tcp128 [8/256/0/180] [ 6659.136903] LNet: Added LNI 192.168.204.142@tcp129 [8/256/0/180] [ 6659.400419] LNet: Added LNI 192.168.204.142@tcp130 [8/256/0/180] [ 6659.661826] LNet: Added LNI 192.168.204.142@tcp131 [8/256/0/180] [ 6659.930107] LNet: Added LNI 192.168.204.142@tcp132 [8/256/0/180] [ 6660.182922] LNet: Added LNI 192.168.204.142@tcp133 [8/256/0/180] [ 6660.456525] LNet: Added LNI 192.168.204.142@tcp134 [8/256/0/180] [ 6660.791052] LNet: Added LNI 192.168.204.142@tcp135 [8/256/0/180] [ 6662.919180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6662.936729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6662.940949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6663.976354] Lustre: 471623:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.142@tcp: can't add NID 192.168.204.142@tcp130: rc = -75 [ 6664.043634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6664.053614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6664.089227] Lustre: lustre-MDT0000: new disk, initializing [ 6664.117423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6664.123149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6665.200775] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6668.894867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6668.914476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6668.949915] Lustre: 471641:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6668.971574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6668.973106] Lustre: Skipped 1 previous similar message [ 6669.007420] Lustre: lustre-MDT0001: new disk, initializing [ 6669.017947] Lustre: 472427:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.142@tcp: can't add NID 192.168.204.142@tcp130: rc = -75 [ 6669.020457] Lustre: 472427:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6669.055992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6669.065766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6669.068571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6670.140299] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6670.643058] Lustre: 472762:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.142@tcp: can't add NID 192.168.204.142@tcp130: rc = -75 [ 6670.645738] Lustre: 472762:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6672.922511] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6673.897820] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6675.810408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6675.835725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6675.971897] Lustre: lustre-OST0000: new disk, initializing [ 6675.973696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6675.999776] Lustre: 473400:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.142@tcp: can't add NID 192.168.204.142@tcp130: rc = -75 [ 6676.002382] Lustre: 473400:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 6676.020370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6677.825732] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6679.227458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6679.230488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6679.266139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6680.541589] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6684.642725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6684.643271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6684.644757] LustreError: Skipped 1 previous similar message [ 6684.648479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6684.650483] Lustre: Skipped 1 previous similar message [ 6688.940081] LustreError: 474233:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6688.980437] Lustre: server umount lustre-OST0000 complete [ 6694.880844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6694.880942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6694.881396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6694.884118] Lustre: Skipped 2 previous similar messages [ 6696.124997] LustreError: 474435:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6696.126807] LustreError: 474435:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6696.152890] Lustre: server umount lustre-MDT0000 complete [ 6697.481043] LustreError: 472426:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773241556 with bad export cookie 15686944724521789856 [ 6697.483943] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6697.484290] LustreError: 472426:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6697.549235] LustreError: 474637:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6697.550974] LustreError: 474637:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6697.595024] Lustre: server umount lustre-MDT0001 complete [ 6700.508544] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6701.922010] Key type lgssc unregistered [ 6702.121871] LNet: 475175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6702.127800] LNetError: 475175:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6702.133526] LNet: Removed LNI 192.168.204.142@tcp [ 6703.199197] LNet: Removed LNI 192.168.204.142@tcp100 [ 6703.201933] LNetError: 475175:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6703.206984] LNetError: 475175:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6703.212048] LNet: Removed LNI 192.168.204.142@tcp101 [ 6704.223192] LNet: Removed LNI 192.168.204.142@tcp102 [ 6704.225990] LNetError: 475175:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6704.231338] LNetError: 475175:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6704.236832] LNet: Removed LNI 192.168.204.142@tcp103 [ 6705.247114] LNet: Removed LNI 192.168.204.142@tcp104 [ 6706.271187] LNet: Removed LNI 192.168.204.142@tcp105 [ 6706.273994] LNetError: 475175:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6706.279363] LNetError: 475175:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 6706.284968] LNet: Removed LNI 192.168.204.142@tcp106 [ 6707.295170] LNet: Removed LNI 192.168.204.142@tcp107 [ 6708.319186] LNet: Removed LNI 192.168.204.142@tcp108 [ 6709.343206] LNet: Removed LNI 192.168.204.142@tcp109 [ 6710.367151] LNet: Removed LNI 192.168.204.142@tcp110 [ 6710.370459] LNetError: 475175:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6710.372706] LNetError: 475175:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 6710.374842] LNet: Removed LNI 192.168.204.142@tcp111 [ 6711.391156] LNet: Removed LNI 192.168.204.142@tcp112 [ 6712.415198] LNet: Removed LNI 192.168.204.142@tcp113 [ 6713.439181] LNet: Removed LNI 192.168.204.142@tcp114 [ 6713.441934] LNet: Removed LNI 192.168.204.142@tcp115 [ 6713.444588] LNet: Removed LNI 192.168.204.142@tcp116 [ 6713.447079] LNet: Removed LNI 192.168.204.142@tcp117 [ 6713.449756] LNet: Removed LNI 192.168.204.142@tcp118 [ 6713.452278] LNet: Removed LNI 192.168.204.142@tcp119 [ 6713.454869] LNet: Removed LNI 192.168.204.142@tcp120 [ 6713.457447] LNet: Removed LNI 192.168.204.142@tcp121 [ 6713.460026] LNet: Removed LNI 192.168.204.142@tcp122 [ 6713.462655] LNet: Removed LNI 192.168.204.142@tcp123 [ 6713.465170] LNet: Removed LNI 192.168.204.142@tcp124 [ 6713.467838] LNet: Removed LNI 192.168.204.142@tcp125 [ 6713.470538] LNet: Removed LNI 192.168.204.142@tcp126 [ 6713.473083] LNet: Removed LNI 192.168.204.142@tcp127 [ 6713.475707] LNet: Removed LNI 192.168.204.142@tcp128 [ 6713.478209] LNet: Removed LNI 192.168.204.142@tcp129 [ 6713.480850] LNet: Removed LNI 192.168.204.142@tcp130 [ 6713.483495] LNet: Removed LNI 192.168.204.142@tcp131 [ 6713.486049] LNet: Removed LNI 192.168.204.142@tcp132 [ 6713.488660] LNet: Removed LNI 192.168.204.142@tcp133 [ 6713.491208] LNet: Removed LNI 192.168.204.142@tcp134 [ 6713.499777] LNet: Removed LNI 192.168.204.142@tcp135 [ 6713.918968] Key type .llcrypt unregistered [ 6713.921193] Key type ._llcrypt unregistered [ 6721.452607] Key type ._llcrypt registered [ 6721.454726] Key type .llcrypt registered [ 6721.485748] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 6727.377327] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6727.781626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6727.791181] alg: No test for adler32 (adler32-zlib) [ 6728.691801] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 6728.808084] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6730.407100] Key type lgssc registered [ 6730.768752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6733.979991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6736.160695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6738.803496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6741.188073] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6743.815812] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 11:06:42 (1773241602) [ 6750.606146] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6751.522183] Key type lgssc unregistered [ 6751.642346] LNet: 479190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6751.644490] LNetError: 479190:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6751.657383] LNet: Removed LNI 192.168.204.142@tcp [ 6751.909401] Key type .llcrypt unregistered [ 6751.910479] Key type ._llcrypt unregistered [ 6758.471620] Key type ._llcrypt registered [ 6758.472628] Key type .llcrypt registered [ 6758.537195] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6758.895601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6758.904616] alg: No test for adler32 (adler32-zlib) [ 6759.786957] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 6759.920645] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6761.543114] Key type lgssc registered [ 6761.877402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6765.618706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6765.652140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6765.659225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6766.742831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6766.753526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6766.787983] Lustre: lustre-MDT0000: new disk, initializing [ 6766.811259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6766.817372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6768.356668] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6772.694766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6772.724477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6772.764681] Lustre: 480360:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6772.807913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6772.811490] Lustre: Skipped 1 previous similar message [ 6772.862084] Lustre: lustre-MDT0001: new disk, initializing [ 6772.913921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6772.936751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6772.943374] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6774.818226] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6777.965524] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6779.573323] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6782.193108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6782.216539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6782.314290] Lustre: lustre-OST0000: new disk, initializing [ 6782.316075] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6782.348428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6782.585631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6782.588505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6782.628692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6784.684983] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6787.847735] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6789.408402] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing lnet_if_list [ 6791.400699] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 6791.410904] LNet: Added LNI 192.168.204.142@tcp42 [8/256/0/180] [ 6798.312003] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6798.316313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6798.321040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6798.323098] Lustre: Skipped 1 previous similar message [ 6803.429859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6803.432145] Lustre: Skipped 2 previous similar messages [ 6803.704136] LustreError: 483738:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6803.768818] Lustre: server umount lustre-OST0000 complete [ 6808.546373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6808.547194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6808.550613] Lustre: Skipped 1 previous similar message [ 6808.553783] Lustre: Skipped 1 previous similar message [ 6811.800245] LustreError: 483945:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6811.804473] LustreError: 483945:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6811.842846] Lustre: server umount lustre-MDT0000 complete [ 6813.469143] LustreError: 480353:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773241672 with bad export cookie 13447929283014446836 [ 6813.476853] LustreError: 480353:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6813.477671] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6813.576745] LustreError: 484147:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6813.579760] LustreError: 484147:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6813.667410] Lustre: server umount lustre-MDT0001 complete [ 6818.514246] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6820.200196] Key type lgssc unregistered [ 6820.443467] LNet: 484686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6820.448874] LNetError: 484686:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6820.453771] LNet: Removed LNI 192.168.204.142@tcp [ 6820.461849] LNet: Removed LNI 192.168.204.142@tcp42 [ 6820.915946] Key type .llcrypt unregistered [ 6820.918297] Key type ._llcrypt unregistered [ 6830.087024] Key type ._llcrypt registered [ 6830.088896] Key type .llcrypt registered [ 6830.163287] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6830.624566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6830.652094] alg: No test for adler32 (adler32-zlib) [ 6831.579055] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 6831.682321] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6833.287200] Key type lgssc registered [ 6833.969404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6838.073263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6838.084739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6839.338545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6841.248792] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6845.469769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6845.663763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6847.615555] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6850.201531] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6851.520453] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6854.450079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6854.636018] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6857.156584] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6859.342208] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6874.281987] loop0: detected capacity change from 0 to 204800000 [ 6874.372712] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6874.379687] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6874.384580] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6874.414154] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6877.063506] loop0: detected capacity change from 0 to 204800000 [ 6877.203487] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6877.210855] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6877.218471] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6877.301659] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6879.677443] loop0: detected capacity change from 0 to 204800000 [ 6882.377799] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6882.418288] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6882.492516] Lustre: fs15246-OST0000: new disk, initializing [ 6882.496663] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 6882.528914] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6885.236436] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6890.375836] loop1: detected capacity change from 0 to 204800000 [ 6893.117903] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6893.158925] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6893.200339] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 6893.223156] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 6893.259620] Lustre: fs15246-MDT0000: new disk, initializing [ 6893.331308] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6893.345251] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6893.395951] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6894.432265] Lustre: 490304:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@00000000ea8e85be x1859378678368512/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1773241769 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 6895.238521] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6903.522708] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6903.528231] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6903.546194] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6907.886818] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6907.895775] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 6911.629059] LustreError: 491427:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 6911.668807] Lustre: server umount fs15246-MDT0000 complete [ 6913.005165] LustreError: 485900:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6913.015807] LustreError: 485900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6916.073318] LustreError: 485902:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6916.199153] LustreError: 491928:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 6916.203829] LustreError: 491928:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6916.232769] Lustre: server umount fs15246-OST0000 complete [ 6921.187866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6921.187940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6921.202871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6921.206822] Lustre: Skipped 1 previous similar message [ 6926.309074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6926.312937] Lustre: Skipped 1 previous similar message [ 6927.217178] LustreError: 492429:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6927.221677] LustreError: 492429:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6927.293808] Lustre: server umount lustre-OST0000 complete [ 6931.427747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6931.428915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6931.435731] Lustre: Skipped 2 previous similar messages [ 6935.251541] LustreError: 492632:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6935.256121] LustreError: 492632:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6935.301608] Lustre: server umount lustre-MDT0000 complete [ 6936.546944] LustreError: 485906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6936.557094] LustreError: 485906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6937.123980] LustreError: 485886:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773241796 with bad export cookie 9743277823953881555 [ 6937.131451] LustreError: 485886:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6937.134078] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6937.327097] Lustre: server umount lustre-MDT0001 complete [ 6942.000822] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6943.617380] Key type lgssc unregistered [ 6943.796738] LNet: 493364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6943.801850] LNetError: 493364:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6943.810649] LNet: Removed LNI 192.168.204.142@tcp [ 6944.298582] Key type .llcrypt unregistered [ 6944.300833] Key type ._llcrypt unregistered [ 6954.010612] Key type ._llcrypt registered [ 6954.012805] Key type .llcrypt registered [ 6954.078864] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6954.404240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6954.434293] alg: No test for adler32 (adler32-zlib) [ 6955.368974] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 6955.504805] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6957.119181] Key type lgssc registered [ 6957.794107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6961.660646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6964.119693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6966.690164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6969.180478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6976.812332] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 6981.479851] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6986.190613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6989.483276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6992.650094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6995.900470] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6999.227302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6999.253174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6999.262161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7000.376861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7000.396743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7000.442954] Lustre: lustre-MDT0000: new disk, initializing [ 7000.472637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7000.483087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7002.070101] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7007.598831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7007.632471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7007.689719] Lustre: 497769:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7007.730608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7007.734534] Lustre: Skipped 1 previous similar message [ 7007.783167] Lustre: lustre-MDT0001: new disk, initializing [ 7007.829769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7007.850949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7007.857684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7009.621307] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7013.366228] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7015.230133] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7018.626404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7018.659089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7018.816666] Lustre: lustre-OST0000: new disk, initializing [ 7018.820852] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7018.872255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7019.387375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7019.393669] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7019.446107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7021.403907] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7025.462618] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7027.332996] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7027.463455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7029.186358] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7029.308976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7034.853612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7034.853726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7034.859452] LustreError: Skipped 1 previous similar message [ 7034.867995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7034.874015] Lustre: Skipped 1 previous similar message [ 7036.340154] LustreError: 500706:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7036.404249] Lustre: server umount lustre-OST0000 complete [ 7038.432600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7038.439188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7038.448773] Lustre: Skipped 1 previous similar message [ 7038.452584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7039.969485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7039.971402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7044.436352] LustreError: 500908:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7044.441347] LustreError: 500908:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7044.486556] Lustre: server umount lustre-MDT0000 complete [ 7046.341153] LustreError: 497762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773241905 with bad export cookie 8384117363744669068 [ 7046.348593] LustreError: 497762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7046.349267] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7046.442559] LustreError: 501111:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7046.446547] LustreError: 501111:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7046.539161] Lustre: server umount lustre-MDT0001 complete [ 7054.915656] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 7057.821414] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7061.258373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7063.272371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7065.516924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7067.520438] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7068.061409] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6825 sec ========== 11:12:07 (1773241927) [ 7068.573247] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:12:07 (1773241927) === [ 7069.585874] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:12:08 (1773241928) === [ 7081.184753] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 7082.027670] Key type lgssc unregistered [ 7082.144262] LNet: 505962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7082.146333] LNetError: 505962:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7082.152412] LNet: Removed LNI 192.168.204.142@tcp [ 7082.402925] Key type .llcrypt unregistered [ 7082.403916] Key type ._llcrypt unregistered